Git Product home page Git Product logo

deeptrade_keras's Introduction

keras version of DeepTrade

Licence(版权声明)

The author is Xiaoyu Fang from China. Please quot the source whenever you use it. This project has key update already. Contact [email protected] to buy a licence.

开源版本对学术应用完全免费,使用时请引用出处;商业应用需要获得授权。鉴于keras已经迁移到tensorflow项目,建议使用DeepTrade的tensorflow版本。

致谢

感谢chenli0830(李辰)贡献的宝贵代码和慷慨捐赠!

Thanks to chenli0830(Chen Li) for his valuable source code and donation!

Experiment

Train Loss: image

Validation Loss: image

A LSTM model using Risk Estimation loss function for trades in market

Introduction

Could deep learning help us with buying and selling stocks in market? The answer could be 'Yes'. We design a solution, named DeepTrade, including history data representation, neural network construction and trading optimization methods, which could maximizing our profit based on passed experience.

In our solution, effective representations are extracted from history data (including date/open/high/low/close/volume) first. Then a neural network based on LSTM is constructed to learn useful knowledges to direct our trading behaviors. Meanwhile, a loss function is elaborately designed to ensure the network optimizing our profit and minimizing our risk. Finaly, according the predictions of this neural network, buying and selling plans are carried out.

Feature Representation

History features are extracted in the order of date. Each day, with open/high/low/close/volume data, invariant features are computed, including rate of price change, MACD, RSI, rate of volume change, BOLL, distance between MA and price, distance between volume MA and volume, cross feature between price and volume. Some of these features could be used directly. Some of them should be normalized. And some should use diffrential values. A fixed length(i.e., 30 days) of feature is extracted for network learning.

Network Construction

LSTM network [1] is effective with learning knowleges from time series. A fixed length of history data (i.e., 30 days) is used to plan trade of next day. We make the network output a real value (p) between 0 and 1, which means how much position (in percent) of the stock we should hold to tomorrow. So that if the rate of price change is r next day, out profit will be p*r. If r is negtive, we lost our money. Therefore, we define a Loss Function (called Risk Estimation) for the LSTM network:

Loss = -100. * mean(P * R)

P is a set of our output, and R is the set of corresponding rates of price change. Further more, we add a small cost rate (c=0.0002) for money occupied by buying stock to the loss function. Then the loss function with cost rate is defined as follows:

Loss = -100. * mean(P * (R - c))

Both of these two loss functions are evaluated in our experiments.

Our network includes four layers: LSTM layer, dense connected layer, batch normalization [3] layer, activation layer. LSTM layer is used to learn knowldges from histories. The relu6 function is used as activation to produce output value.

Trading Plans

Every day, at the time before market close (nearer is better), input history features into the network, then we get an output value p. This p mean an advice of next-day's position. If p=0, we should sell all we have before close. If p is positive, we should keep a poistion of p to next day, sell the redundant or buy the insufficient.

Experimental Results

  If the network goes crazy(overfitting), just restart it. Or, a dropout layer [2] is good idea. Also, larger train dataset will help.

For more demos of the experimental results, visit our website: http://www.deeplearning.xin.

  Experimental Results

Requirements

ta-lib, ta-lib for python, numpy, tensorflow

Bug Report

Contact [email protected] to report any bugs.

Reference

[1] Gers F A, Schmidhuber J, Cummins F, et al. Learning to Forget: Continual Prediction with LSTM[J]. Neural Computation, 2000, 12(10): 2451-2471.

[2] Srivastava N, Hinton G E, Krizhevsky A, et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.

[3] Ioffe S, Szegedy C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[C]. international conference on machine learning, 2015: 448-456.

deeptrade_keras's People

Contributors

happynoom avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

deeptrade_keras's Issues

Adapting this for FOREX on 1 Hour Timeframe

is there a way to adapt your code to work on Forex with 1 hour timeframe . The CSV datafiles is have columns:

DateTime, Open, High , Low , Close , Volume

or

EpochTime, Open, High , Low , Close , Volume

so that it would output buy or sell signal for next hour.

Dropout on input_layer?

Hi @happynoom

Is this model at L25 the best model you tried so far?

Why do you use dropout on the first layer or input_layer? Wouldn't it be more usual to apply dropout on hidden layers?

thanks

something wield on visualization site

On the visualization profit section:
Since 2017.06.26 up to 2017.06.28, the positions are 1s but no profit at all. It is not working properly right?

Ultimate feature and label file

Hi,
Thank you for putting this together. But I am unable to find a way train the model with a new data set. How do I need to prepare the ultimate feature and label files.

some data files are missing?

when running it, there is an error saying can't find the file names with ultimate_feature, ultimate_label and etc.

Here is the error:

  File "/Users/Natsume/miniconda2/envs/tf-experiment/lib/python3.6/site-packages/numpy/lib/npyio.py", line 860, in loadtxt
    fh = iter(open(fname))
FileNotFoundError: [Errno 2] No such file or directory: './ultimate_feature.30'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program

The source code involved:

  23     def read_ultimate(path, input_shape):
  24  ->     ultimate_features = numpy.loadtxt(path + "ultimate_feature." + str(input_sha
  25         ultimate_features = numpy.reshape(ultimate_features, [-1, input_shape[0], in
  26         ultimate_labels = numpy.loadtxt(path + "ultimate_label." + str(input_shape[0
  27         # ultimate_labels = numpy.reshape(ultimate_labels, [-1, 1])
  28         train_set = DataSet(ultimate_features, ultimate_labels)
  29         test_features = numpy.loadtxt(path + "ultimate_feature.test." + str(input_sh
  30         test_features = numpy.reshape(test_features, [-1, input_shape[0], input_shap
  31         test_labels = numpy.loadtxt(path + "ultimate_label.test." + str(input_shape[
  32         # test_labels = numpy.reshape(test_labels, [-1, 1])
  33         test_set = DataSet(test_features, test_labels)
  34         return train_set, test_set

by the way, your code is compatible with keras 2, right?

relu_limited and risk_estimation definition

Hello, nice code, congrats!
I have a question regarding the 2 functions above.
Can relu_limited and risk_estimation be defined inside the python code instead of $PYTHON_DIR/dist-packages/keras/losses.py and $PYTHON_DIR/dist-packages/keras/activations.py ?
I want to run this in a docker container and it would be simpler if I can define them, for instance, in gossip.py.
Please let me know what you think about it.

What does value of pred represent?

I print out pred below and values don't look like the price of prediction of next day.
pred = saved_wp.predict(test_set.images, 1024).
How can I get prediction of next day stock price?

Thanks,

Some questions

  1. Do you train a single model for you whole data set, or one per asset?
  2. How often do you retrain the model?

the best model trained so far?

Hi @happynoom

I have tried the model.30.best come along with your repo. is it the best model with the weights you trained over 3000 epochs?

Have you tried to plot a cumulative return curve with predictions using the best trained model? How good is the curve? Could you upload the image too?

Thanks

Design of model skeleton and feature selection?

Hi @happynoom

I wonder how do you come up with this specific kind of model structure or skeleton. Could you explain the rationality for designing the model in such a structure?

Also, you have chose 61 indicators and normalized them in certain ways. Could you explain why you choose those indicators and why you normalize them in such a way?

Thanks

特征不会用到未来数据吗?

我看到chart.py里有一部分数据,使用到np.mean(),例如norm_volumes = (volumes - numpy.mean(volumes)) / math.sqrt(numpy.var(volumes)),这样训练及测试集一起处理的时候,会用到未来数据吧。如果不一起处理的话,这个mean会很不稳定。

Low accuracy

Hi happynoom,

What was your accuracy when you trained the model. I tried to train it with my dataset and I got a very low accuracy (10-14%). Although with low accuracy, the model performs considerably well.

  • gengstah

特征穿越

在char文件里面,X特征用户第window天的信息(假设window=30, X包含了第30天和30天以前的的close price),p_change 是第30天和第29天的price变化率。 这个穿越太明显了
x = feature_arr[:, p:p + window]

p_change = (close_prices[p + window] - close_prices[p + window - 1]) / close_prices[p + window - 1]

p_change = self.make_label(close_prices[p + window - 1: p + window + self.prospective])

negative loss

Just run model fitting and I am getting:

Train on 86028 samples, validate on 18200 samples
Epoch 1/3000
86028/86028 [==============================] - 16s 185us/step - loss: -0.0560 - acc: 8.8343e-04 - val_loss: -0.0616 - val_acc: 1.0989e-04

Loss is negative, why?

add validation set

Hi @happynoom

you have been using test set as validation set in training.

How about making training set, validation set and test set instead of just training set and test set( which is used as both validation and test set)? It should help to see whether model is really robust or not.

which you use keras version?

python3.5 tensorflow=1.5 keras=2.1,there is error.
MacBook-Pro:DeepTrade_keras boliu$ python gossip.py
/Users/boliu/anaconda3/lib/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
initializing..., learing rate 0.001, n_layers 1, n_hidden 16, dropout rate 0.3.
WARNING:tensorflow:From /Users/boliu/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1242: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
2018-07-25 17:04:10.161405: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
Traceback (most recent call last):
File "gossip.py", line 124, in
make_model([30, 61], 3000, 512, lr=0.001)
File "gossip.py", line 92, in make_model
wp = WindPuller(input_shape=input_shape, lr=lr, n_layers=n_layers, n_hidden=n_hidden, rate_dropout=rate_dropout)
File "/Volumes/BOOTCAMP/dlnlp/DeepTrade_keras/windpuller.py", line 46, in init
metrics=['accuracy'])
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/models.py", line 821, in compile
**kwargs)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 666, in compile
loss_function = losses.get(loss)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/losses.py", line 113, in get
return deserialize(identifier)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/losses.py", line 105, in deserialize
printable_module_name='loss function')
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 159, in deserialize_keras_object
':' + function_name)
ValueError: Unknown loss function:risk_estimation
MacBook-Pro:DeepTrade_keras boliu$

关于数据的组成的问题

@happynoom ,首先,谢谢大佬的分享!投资小白,请教几个问题
1,拜读了一下您的代码,您帮我看一下我的理解对不对:关于数据部分看到dataset下,有26个文件,猜测是26支标的(对应代码),1991年至2007年6464(或不到)天的开,最高,收,最低,交易量的数据。您将这些所有数据,按照日期先后划分,得到一个train,val部分。每个部分由这26支标的所提取的特征(根据selector选取特征),得到输入为[30, 73]的输入,30为窗口大小(每次输入网络某一个标的30天的数据),73为每日特征的长度,预测得到一个P(您在README讲的很清楚了,不再赘述)
2,为什么要选取这26支标的呢?(可以认为都是成分股或者股指?),那老师有试验过,加入更多数据么,结果如何呢?有没有考虑过正负样本的问题呢?
3,关于label的设定,我看到是这样设定的,并不是纯粹的变化率,您能解释一下么?

DeepTrade_keras/chart.py

Lines 31 to 39 in 99511c7

def make_label(self, prices):
ratio = 0.5
decay = 0.9
label = 0.0
for i in range(1, len(prices)):
label = label + (prices[i] / prices[0] - 1.0) * ratio
ratio = ratio * decay
return label / (len(prices) - 1)

谢谢!期待您的回答

Could you provide the website code?

Thank you for providing the concept worthy of my reference. But I encountered the problem of visualizing the chart after running the forecast. I would like to ask here, can you provide the code of the website (like http://www.deeplearning.xin)? If you can provide it, I am really grateful.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.