quantopian / alphalens Goto Github PK
View Code? Open in Web Editor NEWPerformance analysis of predictive (alpha) stock factors
Home Page: http://quantopian.github.io/alphalens
License: Apache License 2.0
Performance analysis of predictive (alpha) stock factors
Home Page: http://quantopian.github.io/alphalens
License: Apache License 2.0
We spent way too long debugging an odd exception:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-59-34a94da3a0f6> in <module>()
2
3 alphalens.tears.create_factor_tear_sheet(factor=signal['signal'],
----> 4 prices=pricing, quantiles=2, periods=(1,))
5 #show_groupby_plots=True)
/usr/local/lib/python2.7/dist-packages/alphalens/plotting.pyc in call_w_context(*args, **kwargs)
41 # sns.set_style("whitegrid")
42 sns.despine(left=True)
---> 43 return func(*args, **kwargs)
44 else:
45 return func(*args, **kwargs)
/usr/local/lib/python2.7/dist-packages/alphalens/tears.pyc in create_factor_tear_sheet(factor, prices, groupby, show_groupby_plots, periods, quantiles, filter_zscore, groupby_labels, long_short, avgretplot, turnover_for_all_periods)
117 mean_monthly_ic = perf.mean_information_coefficient(factor,
118 forward_returns,
--> 119 by_time="M")
120
121 factor_returns = perf.factor_returns(factor, forward_returns, long_short)
/usr/local/lib/python2.7/dist-packages/alphalens/performance.pyc in mean_information_coefficient(factor, forward_returns, group_adjust, by_time, by_group)
133
134 else:
--> 135 ic = (ic.reset_index().set_index('date').groupby(grouper).mean())
136
137 ic.columns = pd.Int64Index(ic.columns)
/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in set_index(self, keys, drop, append, inplace, verify_integrity)
2835 names.append(None)
2836 else:
-> 2837 level = frame[col]._values
2838 names.append(col)
2839 if drop:
/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in __getitem__(self, key)
1995 return self._getitem_multilevel(key)
1996 else:
-> 1997 return self._getitem_column(key)
1998
1999 def _getitem_column(self, key):
/usr/local/lib/python2.7/dist-packages/pandas/core/frame.pyc in _getitem_column(self, key)
2002 # get column
2003 if self.columns.is_unique:
-> 2004 return self._get_item_cache(key)
2005
2006 # duplicate columns & possible reduce dimensionality
/usr/local/lib/python2.7/dist-packages/pandas/core/generic.pyc in _get_item_cache(self, item)
1348 res = cache.get(item)
1349 if res is None:
-> 1350 values = self._data.get(item)
1351 res = self._box_item_values(item, values)
1352 cache[item] = res
/usr/local/lib/python2.7/dist-packages/pandas/core/internals.pyc in get(self, item, fastpath)
3288
3289 if not isnull(item):
-> 3290 loc = self.items.get_loc(item)
3291 else:
3292 indexer = np.arange(len(self.items))[isnull(self.items)]
/usr/local/lib/python2.7/dist-packages/pandas/indexes/base.pyc in get_loc(self, key, method, tolerance)
1945 return self._engine.get_loc(key)
1946 except KeyError:
-> 1947 return self._engine.get_loc(self._maybe_cast_indexer(key))
1948
1949 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4154)()
pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4018)()
pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12368)()
pandas/hashtable.pyx in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:12322)()
KeyError: 'date'
Just to find out that the reason was that the factor index we passed was not UTC localized. Since we only support daily data the timezone does not matter in the first place so we should just cast it to be UTC.
Just noticed that we rename the levels inplace, we should first create a copy to not change the users' data.
To highlight contributions and give credit where it is due.
For demonstrating the capabilities and what it would like if a factor was predictive, we should include made-up data of such a factor with look-ahead bias.
Factor Rank Autocorrelation plot
The plot shows only the daily autocorrelation, but It would be more interesting to plot the autocorrelation after X days, for each X in 'days' (the create_factor_tear_sheet's argument). This suggested behaviour is a superset of the current one because a user can add '1' to 'days' argument to get the current behaviour.
Cumulative Return by Quantile plot
Only the 1 day forward returns graph is plotted. I expected to see plots for each day/period forward return ('days' argument passed to create_factor_tear_sheet ) .
I might have a factor that shows alpha only after X or Z days and I might not be interested to see 1 day forward returns plots, so I would pass days = (X, Z) to create_factor_tear_sheet and I would expect to see the plots for X and Z forward returns. At least it would be nice to have this feature as an option if not the default behaviour.
pyfolio uses gridspec to pull tear sheets into a single fig. Makes tweaking the layout and spacing a little easier. Worth it?
As alphalens is data frequency agnostic, the aggregation by day performed when calculating quantile mean returns doesn't make sense all the time. It could produce better plots for some scenarios, depending on data frequency and data time span, but not for all the possible data given in input to alphalens.
I am suggesting to remove the aggregation by day or to make it optional and in this latter case the aggregation period should be configurable: minute, day, hour, week, whatever.
Also the feature should be applied to all the plots consistently (not only on some of them). Currently it is applied only to plotting.plot_quantile_returns_violin, plotting.plot_cumulative_returns_by_quantile and plotting.plot_mean_quantile_returns_spread_time_series.
http://pandas.pydata.org/pandas-docs/stable/categorical.html
Should make qfactor consume less memory
The docs say that sector_plots is a kwarg in create_factor_tear_sheet()
http://quantopian.github.io/alphalens/alphalens.html#alphalens.tears.create_factor_tear_sheet
However, show_sector_plots is the correct kwarg now.
Top and Bottom Quantile Daily Turnover plot
The plot shows only the daily turnover, but It would be more interesting to plot the turnover after X days, for each X in 'days' (the create_factor_tear_sheet's argument). This suggested behaviour is a superset of the current one because a user can add '1' to 'days' argument to get the current behaviour.
You could feasably pass in any kind of categorical for example country, or industry. By removing the notation sector
we can make the API more extensible.
when looking at the IC histogram plots it seems like each of them is about the same height and thus contain the same number of occurrences. Upon further inspection they all have different sized y axis, this could potentially be misleading
Quantile mean return values change when adding a period (or removing one). Adding/removing a period shouldn't influence the other ones.
Here is part of alphalens output. I ran it two times on the same input, but the second time I added a period of '20'. You can see that in the second output the periods that were already present in the first run change their statistics. It shouldn't happen.
Run 1: periods = (1, 5, 10)
Returns Analysis
1 5 10
Ann. alpha 0.112 0.153 0.232
beta -0.076 0.271 0.260
Mean Period Wise Return Top Quantile (bps) 3.671 4.473 3.340
Mean Period Wise Return Bottom Quantile (bps) -5.745 -6.145 -7.425
Mean Period Wise Spread (bps) 9.416 10.618 10.764
Run 2: periods = (1, 5, 10, 20)
Returns Analysis
1 5 10 20
Ann. alpha 0.112 0.153 0.233 0.287
beta -0.072 0.273 0.261 -0.935
Mean Period Wise Return Top Quantile (bps) 3.653 4.492 3.371 3.023
Mean Period Wise Return Bottom Quantile (bps) -5.744 -6.133 -7.411 -8.493
Mean Period Wise Spread (bps) 9.400 10.622 10.781 11.516
@luca-s the plots plot_quantile_cumulative_avg_return
do not seem to make sense, or at least the error bars dominate. Any idea how to improve that?
The choice of what forward days/period to analyse with create_factor_tear_sheet is somehow arbitrary in my opinion. It would be useful to plot the average cumulative return for each quantile, with standard deviation, over a configurable period of time (this is very similar to the plot you get when running an event study). That would help understanding the average performance of each quantile and decide what days/periods to investigate with the tear sheet.
Please see attached images for a better understanding of what I mean.
Tear-sheet is a bit overloaded currently.
It would be great to have an option to plot the turnover for all quantiles, similar to "Top and Bottom Quantile Daily Turnover" plot but where each quantile has a separate plot.
right now for the IC plots we use the bluish color for the data and the green for the rolling mean. In the returns plots we use green for the data and orange for the rolling mean. it could potentially be confusing, as in it confused me. but i understand we need to differentiate the two plots and colors are one way to do that. perhaps just no overlapping colors
Factor Weighted Cumulative Return plot
Only the 1 day forward returns graph is plotted in 'Factor Weighted Long/Short Portfolio Cumulative Return'. I expected to see plots for each day/period forward return ('days' argument passed to create_factor_tear_sheet) .
I might have a factor that shows alpha only after X or Y days and I might not be interested to see 1 day forward returns plots, so I would pass days = (X, Y) to create_factor_tear_sheet and I would expect to see the plots for X and Y forward returns. At least it would be nice to have this feature as an option if not the default behaviour.
https://github.com/quantopian/alphalens/blob/master/alphalens/performance.py#L162
it should be
demeaned_vals = group - group.mean()
return demeaned_vals / demeaned_vals.abs().sum()
This hard coded range (-0.25, 0.25) can be annoying sometimes.
Here are two graphs that I got running create_factor_tear_sheet with long_short=False (I am not yet sure if this parameter matters).
If you look at quantile 2, the "Mean return by Factor Quantile" graph shows negative mean returns for period 10 and 20, while the "Average cumulative returns by Quantile" graph shows positive mean returns at periods 10 and 20.
I am not sure where the bug is, but it seems there is one.
The forward returns used for plotting are converted to returns relative to mean. While this is a very clever choice (the code is a constant surprise of many smart ideas, well done!) it implies a long short portfolio and that is not always the case. One might be interested to see if their signal can be profitable in a long (or short) only portfolio over a particular time period. So, why not making the demeaning optional?
Alphalens is shuch a great swiss knife, so why using the blade only? :D
Many thanks!
create_factor_tear_sheet
accept the same data format as all theformat_input_data
(seecompute_mean_returns_spread
and mean_return_by_quantile
) thatstd_err
, which is sometimes a bool and sometimes a series.)tears.py
, despite the fact thatmean_information_coefficient
is uncovered when len(grouper) == 0
.factor_returns
is uncovered when long_short
is False.factor_alpha_beta
is uncovered when factor_daily_returns
is None. (Docompute_forward_returns
is uncovered when filter_zscore
isn't passedmean_return_by_quantile
, orcompute_mean_returns_spread
, or format_input_data
. (And there arecompute_mean_returns_spread
andmean_return_by_quantile
: See Below).tears.py
from plotting import *
: Please don't do this. It immediately breaks anycreate_factor_tear_sheet
Most of the comments here apply to the parameters with the same names elsewhere:
sector_plots
: I would probably call this show_sector_plots
to be clearerfilter_zscore
only be an int
? It seems like this would bedays
is annotated as list, but the default is a tuplesequence[int]
. More seriously, if I pass in my own list,performance.py
Several functions in this file accept an argument named std_err
. Sometimes
it's a bool indicating whether an optional value should be computed. Other
times, it's a Series of computed values. Most of the boolean invocations are
subtly broken at their only call-sites, so I'd argue we should just remove
them. If we decide they're really important, I'd rename the bools to something
like compute_std_err
, to be clearer that it's a toggle and not the computed
values themselves.
factor_alpha_beta
quantize_factor
list
means alist
when it's not what you actuallyalphalens/performance.py:144: A list of equities and their factor values indexed by date. alphalens/performance.py:182: A list of equities and their factor values indexed by date. alphalens/performance.py:230: A list of equities and their factor values indexed by date. alphalens/performance.py:266: A list of equities and their N day forward returns where each column contains the N day forward returns. alphalens/utils.py:154: A list of equities and their factor values indexed by date. alphalens/utils.py:163: A list of equities and their sectors. alphalens/utils.py:178: A list of equities and their factor values indexed by date, docs/source/conf.py:99: # A list of ignored prefixes for module index sorting.
factor_rank_autocorrelation
compute_mean_returns_spread
tears.py
, where we do:mean_ret_spread_quant, std_spread_quant = perf.compute_mean_returns_spread( mean_ret_quant_daily, quantiles, 1, std_err=std_quant_daily)
std_err
.mean_return_by_quantile
compute_mean_returns_spread
: we're alwaysutils.py
format_input_data
ticker_sector = {...}
: it'd be nice if there was some context/explanationfrom pandas.io.data import DataReader
: pandas.io.data is deprecated. Wefor ticker in ticker_sector.keys()
: we don't need .keys()
here.While using quantiles is the most straightforward choice there are use cases where it would be much more reasonable to classify the factor values using equal interval bins.
An example is a common scenario where a factor produces positive and negative values (e.g. in -1, 1 range) and alphalens should be able to calculate statistics on two distinct group: the positive values group and the negative values one. Currently this is not possible if the positive and negative values are not equally balanced (50% of positive values and 50% of negative values).
So I am proposing to have two way of classifying the factor values:
documentation of factor_alpha_beta mentions that it should return t-value, but function factor_alpha_beta did not add t-value in the returned dataframe. And in the function, it actually computes the t-value from OLS. Forgot to add a new row ? Or not ready to commit?
While the "mean daily return by quantile" plot tells us how well the factor differentiates forward returns across the signal/factor values, the shown results depend strongly on the number of quantiles used. So it would be nice to add a distribution plot of returns vs signal/factor to properly see the relationship between returns and signal
Just to give an idea of what I mean, here are some pictures.
I noticed the daily forward returns are calculated as the forward returns divides by the forward period (see utils.compute_forward_returns) .
I don't exactly understand the reason so I might be wrong, but shouldn't the daily returns calculated as:
# let's pretend those are the variables
# 'days' is our period
# 'fwd_ret' is the return after the period
# the current implementation does something like the following (what does this value mean?)
daily_returns = fwd_ret / days
# while I am suggesting to calculate the daily returns as the value the returns would have
# had every single day for the whole period if they had grown at a steady rate
daily_returns = (fwd_ret+1)**(1./days) - 1
When I use pip to install alphalens, it shows
Could not find a version that satisfies the requirement alphalens (from versions: )
No matching distribution found for alphalens
Here we can see the data passed to plot_quantile_returns_violin is not the same passed to plot_quantile_returns_bar (mean_ret_quant_daily vs mean_ret_quantile).
It seems mean_ret_quant_daily was meant to be used by plot_cumulative_returns_by_quantile only. So I believe that's a typo, but I might be wrong.
And also need to fix the bug in factor_alpha_beta function, where it requires factor_returns as an argument, but it uses the same name as function factor_returns. So if factor_returns = None, it would raise an error that None cannot be called.
Can be replicated in this notebook.
Add a simple test that runs the tear-sheet and makes sure it does not raise an exception.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.