jealous / stockstats Goto Github PK

Hi,

kdj is terribly slow, probably because the result is created step by step with list.append().
I did some speed tuning which gives about 60 times speed up, code see below.
I tried to vectorize as much as possible and to have as little array accesses as possible.
Works only with python 3 because I used nonlocal in the inline function.
I guess it could be made even faster with something like scipy.signal.lfilter.

    @classmethod
    def _get_kdjk(cls, df, n_days):
        """ Get the K of KDJ
        Overwrite stockstats.StockDataFrame methods to speed up.

        K ＝ 2/3 × (prev. K) +1/3 × (curr. RSV)
        2/3 and 1/3 are the smooth parameters.
        :param df: data
        :param n_days: calculation range
        :return: None
        """
        rsv_column = 'rsv_{}'.format(n_days)
        k_column = 'kdjk_{}'.format(n_days)

        KDJ_PARAM0, KDJ_PARAM1 = cls.KDJ_PARAM
        prev_k = 50.0

        def __add(t):
            nonlocal prev_k
            prev_k = KDJ_PARAM0 * prev_k + t
            return prev_k

        df[k_column] = [__add(t) for t in KDJ_PARAM1 * df.get(rsv_column).as_matrix()]

    @classmethod
    def _get_kdjd(cls, df, n_days):
        """ Get the D of KDJ

        D = 2/3 × (prev. D) +1/3 × (curr. K)
        2/3 and 1/3 are the smooth parameters.
        :param df: data
        :param n_days: calculation range
        :return: None
        """
        k_column = 'kdjk_{}'.format(n_days)
        d_column = 'kdjd_{}'.format(n_days)

        KDJ_PARAM0, KDJ_PARAM1 = cls.KDJ_PARAM
        prev_d = 50.0

        def __add(t):
            nonlocal prev_d
            prev_d = KDJ_PARAM0 * prev_d + t
            return prev_d

        df[d_column] = [__add(t) for t in KDJ_PARAM1 * df.get(k_column).as_matrix()]

WR results a positive number. Should the formula be multiplied with -100 rather than 100?

Looking at the various definitions I can find on the MACD histogram, Everything I can find simply defines it as the difference between the MACD and the 9 period signal line. Various tools like tradingview, cryptowatch, etc. use this as the definition, as does investopedia.

However in comparing the data I get from stockstats, it appears that a doubling modifier is added:

df['macdh'] = 2 * (df['macd'] - df['macds'])

Why is that? I can't find anything to support such a formula. If you agree, I'm happy to submit my PR.

I think this would be an interesting feature that I'd be willing to help out with.

Ideally you could call a function similar to df.get_stock_stats() that returns a data frame with every single indicator.

any advice?

In def _get_s, change the last line of code (StockDataFrame.set_nan(df[shifted_key], shift)) to

a=df[shifted_key].copy()
StockDataFrame.set_nan(a, shift)
df[shifted_key]=a.copy()

Same for def _get_d, change the last line of code (StockDataFrame.set_nan(df[column_name], shift)) to

a=df[column_name].copy()
StockDataFrame.set_nan(a, shift)
df[column_name]=a.copy()

Should be the same for def _get_p, but I don't know which indicators are using this function, so I didn't test it. should change the last two lines of code:

StockDataFrame.set_nan(indices, shifts)
df[column_name] = indices

Also suggest to provide a flag to disable MACDH warning.

cci

cci sometimes 'inf'

hi,

is it possible to add support for the TEMA? It should be pretty straight forward from my understanding.

https://www.forextraders.com/forex-education/forex-technical-analysis/triple-exponential-moving-average-the-tema-indicator/

kind regards

Hello!
Question regarding how RSI 10 is calculated ... When providing a pandas dataframe

df = pd.DataFrame(list)
recs = StockDataFrame.retype(df)
recs['rsi_10']

If I print out the dataframe:

close 970.27
id 34156
volume 184.7046024000001694
close_-1_s NaN
close_-1_d NaN
rs_10 NaN
rsi_10 NaN
Name: 0, dtype: object
close 970.49
id 34157
volume 148.3538014100001874
close_-1_s 970.27
close_-1_d 0.22
rs_10 inf
rsi_10 100
Name: 1, dtype: object
close 967.95
id 34158
volume 108.4361391200002487
close_-1_s 970.49
close_-1_d -2.54
rs_10 0.0779528
rsi_10 7.23156
Name: 2, dtype: object
close 968.14
id 34159
volume 125.0943135599999323
close_-1_s 967.95
close_-1_d 0.19
rs_10 0.161067
rsi_10 13.8724
...

The RSI 10 is already being calculated in the third row - isn't RSI 10 supposed to take the first 10 rows to calculate the initial RSI?

Thanks! And sorry if this is not the place for questions, I searched on Stackoverflow and couldn't find any topic or keyword for stockstats

very useful package, thanks!

What I'm missing is RSI (see https://en.wikipedia.org/wiki/Relative_strength_index)
and a bit more verbose documentation ;-)

Would it be hard to implement RSI?

When writing:
ss = Sdf.retype(dataf)
dataf['rsi_14'] = ss['rsi_14']

I get:

D:\PY\ANACONDA3\lib\site-packages\pandas\core\indexing.py:179: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self._setitem_with_indexer(indexer, value)

Am I doing something wrong ?

was not related. solved

I get the following error when I try to retrieve rsi_14. My dataframe looks like this:

          open      high       low     close    volume

date
20191113 261.1300 264.7800 261.0700 264.4200 20815595
20191112 261.5500 262.7900 260.9200 261.9600 21826100
20191111 258.3000 262.4700 258.2800 262.2000 20455300
20191108 258.6900 260.4400 256.8500 260.1400 17496600
20191107 258.7400 260.3500 258.1100 259.4300 23735100
... ... ... ... ... ...
20190701 203.1700 204.4900 200.6500 201.5500 27253000
20190628 198.6800 199.4950 197.0500 197.9200 31110600
20190627 200.2900 201.5700 199.5700 199.7400 20899700
20190626 197.7700 200.9900 197.3494 199.8000 26067500
20190625 198.4300 199.2600 195.2900 195.5700 21070300

Hi,

Is there an Adaptive Moving Average (AMA) indicator in stockstats, which means a moving median for a said period?

Thanks

'StockDataFrame' object has no attribute 'get_today'

anaconda3/lib/python3.6/site-packages/stockstats.py:387: FutureWarning: Currently, 'apply' passes the values as ndarrays to the applied function. In the future, this will change to passing it as Series objects. You need to specify 'raw=True' to keep the current behaviour, and you can pass 'raw=False' to silence this warning
lambda x: np.fabs(x - x.mean()).mean())

Obviously, this is a quick fix. Has anyone else encountered it and is there a patch so we can pip install -u stockstats to resolve this issue?

Talking about the ADX index, I checked several programs and found the ADXs are all different. For stockstats, the ADX looks a bit higher than others. I am quite confused about this index calculation.

I use this package. Ultimately I want to train neural nets with the data. Would really like to see an incorporation of the basic datas: P/E , EPS, P/B, PEG, etc. I can get this from yahoo finance (I think). But would like a one stop shop for all may data in a common format. Willing to help, but would need to be brought up to speed with the data frames.

http://pythondata.com/stockstats-python-module-various-stock-market-statistics-indicators/

Hey,

Wondering if somebody could help a guy out in getting this installed and setup up to use? I'm pretty new at this so any advice would be wonderful.

I have python installed, downloaded the files and I researched that I can get the "pandas.DataFrame" through Anaconda. Not sure where to go from here.

Thanks

First, thank you very much for the great work!

It seems that the counting only applies to a number of days in the past. Would it be nice to have the capability to count forward? Say, 'close_3.5_ge_-5_c' means to count the number of times in the future 5 days where close price is more than $3.5.

Hope this makes sense. Thanks!

Hello everyone. I am trying to use Stockstats to analyze data from Cryptocurrencies which have 8 decimals positions.

The results only use a precision of 6 decimals. Is there a way to configure stockstats to use more decimals?

If I convert the original data to float it reduces the precision to 6.

Imported Data (Pandas):
datetime open high low close
0 2018-01-22 21:32:00 0.00000632 0.00000633 0.00000631 0.00000632
1 2018-01-22 21:33:00 0.00000631 0.00000632 0.00000631 0.00000631
2 2018-01-22 21:34:00 0.00000631 0.00000632 0.00000631 0.00000632

dtype: object
datetime datetime64[ns]
open object
high object
low object
close object

Results using Stockstats of Bollinger Bands:
close_20_mstd boll boll_ub boll_lb close_13_ema close_34_ema
0 NaN 0.000006 NaN NaN 0.000006 0.000006
1 7.071068e-09 0.000006 0.000006 0.000006 0.000006 0.000006
2 5.773503e-09 0.000006 0.000006 0.000006 0.000006 0.000006
3 5.000000e-09 0.000006 0.000006 0.000006 0.000006 0.000006

close_20_sma float64
close_20_mstd float64
boll float64
boll_ub float64
boll_lb float64
close_13_ema float64
close_34_ema float64

I want to use other values for example in bollinger I would use 8 periods... how is the correct way to do it?

just do a "Sdf.BOLL_PERIOD = 8" ?

Is there a reason why the MACD histogram value is doubled? From the resources I've seen and the trading client I use, it seems to be calculated as only the difference instead of double the difference.

(And thanks for your work on this library! It has saved me quite a bit of work on a side project of mine.)

this is more a feature request.

Basically, it's really easy to pull these bulk download sheets of daily stock data from sources like quandl that have the ticker as a column, then the date for each line of data. If I want to use stockstats properly to calculate DMI or whatever, it looks like I need to use a for loop to iterate through each stock one by one. It'd be much cooler if I could simply use 'retype()' on a dataframe indexed with both ticker and date. I am actually able to do this now (surprisingly), and it runs very quickly even over 15k tickers spanning several years, but the problem is the 14 period moving averages and things like that seem to use the previous stock's data. Since I'm looking at several years of data, this is only an issue for stocks that IPO'd recently, but it would still be nice if the code had some 'if' statements or whatever to check whether a line should be included in the moving average calculations when used this way. It'd be a cool way to make this even easier to use, in my opinion.

Just came across your repo and it looks very promising for what I'm doing! Really impressive that you generated all of these indicators.

It would be great to be able to generate indicators for different frequencies - month, week, hour, minute. Have you looked into this at all? This would be useful for day trading. Then users could do:

dataframe.get('rsi_1_H')
dataframe.get('rsi_30_min')
dataframe.get('rsi_15_min')

It could be a backwards compatible change in that if no frequency is specified, daily frequency is used. Otherwise specify the frequency using the Pandas offset alias format.

The data frame would have to have the highest frequency asked for (minute-level for minute frequency, etc.). The library would have to be able to determine the frequency of the data,
and resample the data to generate indicators for the higher time series.

I can look into this more also if you can offer thoughts on how feasible you think this is and what pieces of code I should look at.

Hi, thanks for your contribution.
I was running the package and failed to get EMA with both

stock.get("ema")
and
stock.get("EMA")

May I know how you get this?

I was trying to extract SMA values from the dataframe, but the code is throwing error. Pasted below is my code and error details.

results = Sdf.retype(get_history(symbol =AAPL,start=date(2018,03,27),end=date(2018,06,05),index=False))
data_sma=results['open_2_sma']
print (data_sma[-1])

Error message:
No handlers could be found for logger "stockstats"

Traceback (most recent call last):
File "C:\Python27\Practises\Investing\StockStats\Training.py", line 35, in
print ("SMA:",results['data_sma'][-1])
File "C:\Python27\lib\site-packages\stockstats.py", line 929, in getitem
super(StockDataFrame, self).getitem(item))
File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 2139, in getitem
return self._getitem_column(key)
File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 2146, in _getitem_column
return self._get_item_cache(key)
File "C:\Python27\lib\site-packages\pandas\core\generic.py", line 1842, in _get_item_cache
values = self._data.get(item)
File "C:\Python27\lib\site-packages\pandas\core\internals.py", line 3843, in get
loc = self.items.get_loc(item)
File "C:\Python27\lib\site-packages\pandas\core\indexes\base.py", line 2527, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas_libs\index.pyx", line 117, in pandas._libs.index.IndexEngine.get_loc
File "pandas_libs\index.pyx", line 139, in pandas._libs.index.IndexEngine.get_loc
File "pandas_libs\hashtable_class_helper.pxi", line 1265, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas_libs\hashtable_class_helper.pxi", line 1273, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'data_sma'

Stockstats is very powerful in computing statistics on daily data. Can it also work with intraday data, like 15-minute? Thanks for the great work!

Since this module already has TR, ATR; it should be simple to calculate supertrend.

Hi,
When I use the rsi indicator pandas gives me this warning:

/home/martin/PycharmProjects/test/venv/lib/python3.8/site-packages/pandas/core/indexing.py:670: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_with_indexer(indexer, value)

The code:

import stockstats
import pandas as pd


def test(csv, indicator):
    sdf = stockstats.StockDataFrame.retype(pd.read_csv(csv))
    value = list(sdf.loc[:, indicator])


test("BTC-USD.csv", "rsi_14")

BTC-USD.csv:

Data,Low,High,Open,Close,Volume
1581857700,9906.32,9917.98,9915.37,9917.98,3.66869252
1581857760,9917.45,9926.23,9917.45,9924.99,2.95763819
1581857820,9925.13,9927,9925.13,9926.58,0.2517662
1581857880,9922.74,9926.93,9926.59,9923.85,0.16580884
1581857940,9922.33,9923.85,9923.85,9923.8,0.75820216
1581858000,9919.35,9923.8,9923.8,9920.32,3.27751825
1581858060,9910.68,9919.35,9919.35,9910.68,0.27416278
1581858120,9906.32,9914.46,9910.68,9906.46,0.77074594
1581858180,9906.32,9906.39,9906.38,9906.33,0.09115752
1581858240,9891,9905,9905,9891,3.82049405
1581858300,9891,9905.95,9893.8,9903.12,5.00440502
1581858360,9900.53,9910.95,9903.12,9905.35,5.5605933
1581858420,9901.92,9903.36,9903.36,9903.19,0.21576107
1581858480,9903.19,9903.64,9903.19,9903.56,0.34238175
1581858540,9897.87,9901.04,9901.04,9898.3,4.69393022
1581858600,9884.31,9898.64,9898.63,9887.94,3.21781626
1581858660,9884,9899.49,9884.01,9898.27,7.51834045
1581858720,9871.02,9899.49,9898.28,9882.2,34.62721212
1581858780,9875.24,9891.88,9882.21,9885,4.54068467
1581858840,9883.78,9886.09,9885.01,9885.41,2.93844096
1581858900,9885.64,9908.07,9885.64,9903.08,6.1523356
1581858960,9896.28,9900,9900,9896.28,1.45623205
1581859020,9896.28,9907.83,9896.28,9907.83,3.23573508
1581859080,9905.94,9907.85,9905.97,9905.95,0.35000316
1581859140,9897.68,9905.95,9905.95,9897.68,0.9411693
1581859200,9895,9896.04,9895.27,9895.11,0.17042081
1581859260,9886.43,9895.11,9895.11,9886.43,1.52060855
1581859320,9878.51,9889.2,9886.42,9886.32,7.34562765
1581859380,9878.5,9888.37,9888.37,9880.19,1.2896627

pandas is 1.0.1 and stockstats is 0.3.1
I know it's just a warning, but I hope it would be fixed.

Why is the MACD histogram being multiplied by 2?:

df['macdh'] = 2 * (df['macd'] - df['macds'])

Every time I see it explained, it's defined as the difference between the MACD line and the signal line.
Thanks

MACD should not have hardcoded EMA periods- they need to be variable inputs.

Infact none of these functions should have hardcoded periods, it's severely limits the usefulness of an otherwise stellar wrapper.

hi there. is there any indicator that can be used instead of VWAP (Volume-weighted average price) ?

Could you please explain to me what something like this would do:
close_-3,-1,+2_p
open_-1_d_-1,-3_p

I looked at your tests, the main .py file and for the life of mine, can't figure out what _p does.

Please advise.

does MACD support any other EMA besides (12,26,9)? maybe (5,35,5)?

also, is PPO supported? (http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:price_oscillators_ppo)

thanks, awesome lib!

Hi,

there are cases where the kdj columns are all NaN. This seems to be related to rsv_9 beeing NaN.

Here is sample data (the first 20 lines of ASML stock quote) which shows the problem:

date        adj_close     close      high       low      open     volume
2010-01-01   29.28827  23.99998  23.99998  23.99998  23.99998        0.0
2010-01-04   29.60560  24.26000  24.31999  23.89002  23.90003  1563900.0
2010-01-05   29.66047  24.30497  24.62498  24.05996  24.11501  1550300.0
2010-01-06   29.97780  24.56500  24.56500  24.18000  24.20503  1133900.0
2010-01-07   29.42866  24.11501  24.45004  23.80001  24.45004  2648700.0
2010-01-08   28.44013  23.30497  24.08498  23.27502  23.99998  3064200.0
2010-01-11   27.37239  22.43002  23.45497  22.43002  23.31999  4640500.0
2010-01-12   27.82389  22.80001  22.92998  22.60004  22.65001  3098000.0
2010-01-13   28.28762  23.18000  23.39999  22.75003  22.80001  3732600.0
2010-01-14   28.26319  23.15998  23.59996  23.09499  23.46998  1851800.0
2010-01-15   27.89709  22.85999  23.43002  22.69498  23.28996  2738400.0
2010-01-18   27.93985  22.89503  22.98504  22.62499  22.90003  1132900.0
2010-01-19   27.67129  22.67496  22.91997  22.62499  22.80501  2392200.0
2010-01-20   28.53165  23.37997  23.87000  22.87501  22.98997  6490400.0
2010-01-21   29.00759  23.76998  24.06997  23.65001  23.74503  4068900.0
2010-01-22   28.53165  23.37997  23.82003  23.33500  23.53998  3842600.0
2010-01-25   27.78725  22.76998  23.18501  22.69998  22.70999  3091000.0
2010-01-26   28.34861  23.22998  23.37004  22.81002  22.99998  2716300.0
2010-01-27   28.12290  23.04502  23.13503  22.71500  23.08999  2130900.0
2010-01-28   27.86656  22.83497  23.73001  22.83497  23.44003  3445800.0

if rsv_9 is calculated with stockstats, the first value is NaN. This leads to all kdj columns beeing NaN as well. I guess this is due to a division by zero error in _get_rsv in line 251:

df[column_name] = ((df['close'] - low_min) /
                   (high_max - low_min).astype('float64') * 100)  

How should the code be modified so that this bug doesn't appear?
(or what's the correct value for rsv if high_max - low_min == 0?)

are TEMA and TRIX adjustable like other metrics such as .get('tema_20') || .get('trix_35')

Couldn't find on stackoverflow or any other forums so I am at the source : D

Awesome lib!

Being able to set the standard deviation and SMA amounts when calculating Bollinger Bands would be helpful to eliminate noise using a Mean Revision strategy (see https://marketfy.com/content/74202-how-to-trade-mean-reversion-setups for an example). Would be happy to work on this.

the retype method will throw 'tuple has no attribute lower' error. the Pandas version is 0.22.0

A weighted moving average is calculated as x = (n*Ct + (n-1)Ct-1 + ... + 1Ct-n) / (n+n-1+n-2+...+1)

my code:

    @classmethod
    def get_atr(cls, candles):
        stock = StockDataFrame.retype(TCandle.to_df(candles))
        return list(stock.get('atr'))

I got the following warning:

  /Users/yurenji/.conda/envs/tangle/lib/python3.6/site-packages/pandas/core/indexing.py:189: SettingWithCopyWarning: 
  A value is trying to be set on a copy of a slice from a DataFrame
  
  See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
    self._setitem_with_indexer(indexer, value)

-- Docs: http://doc.pytest.org/en/latest/warnings.html

I tried 'macdh' and 'sma', they don't have this issue.

Hi! I was wondering if there's a method to append new data. I would like to do something like that:

new_frame = pd.DataFrame(data=[[date, open, high, low, close]], columns=['DateTime', 'Open', 'High', 'Low', 'Close'])
Data = Data.append(new_frame, ignore_index=True)

This code will throw you an exception because you have to marge the different columns. But, it would be great if we just required to add the OHLC data so that the StockDataFrame automatically recalculates the statistics and indicators.
Are there different alternatives to append new data?

Thanks in advance,

Do you have any plans to implement ATR?

Hi,

Pandas recently released 1.0.0 and since then, rsi calculations seem to run into KeyError for some reason.

Here's very simple sample code:

#!/usr/bin/env python3

import pandas_datareader.data as web
from stockstats import StockDataFrame

spx = web.DataReader('^GSPC', 'yahoo')
dataframe = StockDataFrame.retype(spx)
print(dataframe['rsi_14']['2020-01-31'])

With pandas 0.25.3, it works fine:

Successfully installed certifi-2019.11.28 chardet-3.0.4 idna-2.8 int-date-0.1.8 lxml-4.5.0 numpy-1.18.1 pandas-0.25.3 pandas-datareader-0.8.1 python-dateutil-2.8.1 pytz-2019.3 requests-2.22.0 six-1.14.0 stockstats-0.3.0 urllib3-1.25.8
$ ./test.py
venv/lib/python3.6/site-packages/pandas/core/indexing.py:205: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_with_indexer(indexer, value)
43.43522572721968

However, with pandas 1.0.0, KeyError occurs:

Successfully installed certifi-2019.11.28 chardet-3.0.4 idna-2.8 int-date-0.1.8 lxml-4.5.0 numpy-1.18.1 pandas-1.0.0 pandas-datareader-0.8.1 python-dateutil-2.8.1 pytz-2019.3 requests-2.22.0 six-1.14.0 stockstats-0.3.0 urllib3-1.25.8
$ ./test.py 
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self._setitem_with_indexer(indexer, value)
Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2646, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1622, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'rsi_14'

The problem could be on pandas side, though they bumped the major version so I suspect stockstats might want to support retyping of pandas 1.0.0 and on as well.

Hi,
What a wonderful tool you have created!
I'm facing the following error using cross line indicator:

_ = chart['close_x_close_30_ema']

C:\Users\grafeer\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy\lib\function_base.py in diff(a, n, axis)

   1924         return diff(a[slice1]-a[slice2], n-1, axis=axis)
   1925     else:
-> 1926         return a[slice1]-a[slice2]
   1927 
   1928 

TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead.

I'm using numpy 1.13.0 version

I have stockstats installed with pip but I get the abovementioned error when i run:

from stockstats import StockDataFrame as Sdf

Any ideas?