when running it, there is an error saying can't find the file names with ultimate_feature, ultimate_label and etc.

Here is the error:

  File "/Users/Natsume/miniconda2/envs/tf-experiment/lib/python3.6/site-packages/numpy/lib/npyio.py", line 860, in loadtxt
    fh = iter(open(fname))
FileNotFoundError: [Errno 2] No such file or directory: './ultimate_feature.30'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program

The source code involved:

  23     def read_ultimate(path, input_shape):
  24  ->     ultimate_features = numpy.loadtxt(path + "ultimate_feature." + str(input_sha
  25         ultimate_features = numpy.reshape(ultimate_features, [-1, input_shape[0], in
  26         ultimate_labels = numpy.loadtxt(path + "ultimate_label." + str(input_shape[0
  27         # ultimate_labels = numpy.reshape(ultimate_labels, [-1, 1])
  28         train_set = DataSet(ultimate_features, ultimate_labels)
  29         test_features = numpy.loadtxt(path + "ultimate_feature.test." + str(input_sh
  30         test_features = numpy.reshape(test_features, [-1, input_shape[0], input_shap
  31         test_labels = numpy.loadtxt(path + "ultimate_label.test." + str(input_shape[
  32         # test_labels = numpy.reshape(test_labels, [-1, 1])
  33         test_set = DataSet(test_features, test_labels)
  34         return train_set, test_set

by the way, your code is compatible with keras 2, right?

@happynoom why do you choose relu_limited over sigmoid?

It looks like relu_limited cut off things, whereas sigmoid still consider very negative and very large positive numbers.

Have you tried both and found relu_limited outperform sigmoid?

在char文件里面，X特征用户第window天的信息（假设window=30， X包含了第30天和30天以前的的close price），p_change 是第30天和第29天的price变化率。 这个穿越太明显了
x = feature_arr[:, p:p + window]

p_change = (close_prices[p + window] - close_prices[p + window - 1]) / close_prices[p + window - 1]

p_change = self.make_label(close_prices[p + window - 1: p + window + self.prospective])

@happynoom ,首先，谢谢大佬的分享！投资小白，请教几个问题
1，拜读了一下您的代码，您帮我看一下我的理解对不对：关于数据部分看到dataset下，有26个文件，猜测是26支标的（对应代码），1991年至2007年6464（或不到）天的开，最高，收，最低，交易量的数据。您将这些所有数据，按照日期先后划分，得到一个train,val部分。每个部分由这26支标的所提取的特征（根据selector选取特征），得到输入为[30, 73]的输入，30为窗口大小（每次输入网络某一个标的30天的数据），73为每日特征的长度，预测得到一个P（您在README讲的很清楚了，不再赘述）
2，为什么要选取这26支标的呢？（可以认为都是成分股或者股指？），那老师有试验过，加入更多数据么，结果如何呢？有没有考虑过正负样本的问题呢？
3，关于label的设定，我看到是这样设定的，并不是纯粹的变化率，您能解释一下么？

谢谢！期待您的回答

Thank you for providing the concept worthy of my reference. But I encountered the problem of visualizing the chart after running the forecast. I would like to ask here, can you provide the code of the website (like http://www.deeplearning.xin)? If you can provide it, I am really grateful.

Hi @happynoom

you have been using test set as validation set in training.

How about making training set, validation set and test set instead of just training set and test set( which is used as both validation and test set)? It should help to see whether model is really robust or not.

Hi @happynoom

Is this model at L25 the best model you tried so far?

Why do you use dropout on the first layer or input_layer? Wouldn't it be more usual to apply dropout on hidden layers?

thanks

python3.5 tensorflow=1.5 keras=2.1,there is error.
MacBook-Pro:DeepTrade_keras boliu$ python gossip.py
/Users/boliu/anaconda3/lib/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
initializing..., learing rate 0.001, n_layers 1, n_hidden 16, dropout rate 0.3.
WARNING:tensorflow:From /Users/boliu/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1242: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
2018-07-25 17:04:10.161405: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
Traceback (most recent call last):
File "gossip.py", line 124, in
make_model([30, 61], 3000, 512, lr=0.001)
File "gossip.py", line 92, in make_model
wp = WindPuller(input_shape=input_shape, lr=lr, n_layers=n_layers, n_hidden=n_hidden, rate_dropout=rate_dropout)
File "/Volumes/BOOTCAMP/dlnlp/DeepTrade_keras/windpuller.py", line 46, in init
metrics=['accuracy'])
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/models.py", line 821, in compile
**kwargs)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 666, in compile
loss_function = losses.get(loss)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/losses.py", line 113, in get
return deserialize(identifier)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/losses.py", line 105, in deserialize
printable_module_name='loss function')
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 159, in deserialize_keras_object
':' + function_name)
ValueError: Unknown loss function:risk_estimation
MacBook-Pro:DeepTrade_keras boliu$

I print out pred below and values don't look like the price of prediction of next day.
pred = saved_wp.predict(test_set.images, 1024).
How can I get prediction of next day stock price?

Thanks,

Hi,
Thank you for putting this together. But I am unable to find a way train the model with a new data set. How do I need to prepare the ultimate feature and label files.

is there a way to adapt your code to work on Forex with 1 hour timeframe . The CSV datafiles is have columns:

DateTime, Open, High , Low , Close , Volume

or

EpochTime, Open, High , Low , Close , Volume

so that it would output buy or sell signal for next hour.

Hi happynoom,

What was your accuracy when you trained the model. I tried to train it with my dataset and I got a very low accuracy (10-14%). Although with low accuracy, the model performs considerably well.

• gengstah

Hello, nice code, congrats!
I have a question regarding the 2 functions above.
Can relu_limited and risk_estimation be defined inside the python code instead of $PYTHON_DIR/dist-packages/keras/losses.py and $PYTHON_DIR/dist-packages/keras/activations.py ?
I want to run this in a docker container and it would be simpler if I can define them, for instance, in gossip.py.
Please let me know what you think about it.

Hi @happynoom

I have tried the model.30.best come along with your repo. is it the best model with the weights you trained over 3000 epochs?

Have you tried to plot a cumulative return curve with predictions using the best trained model? How good is the curve? Could you upload the image too?

Thanks

Just run model fitting and I am getting:

Train on 86028 samples, validate on 18200 samples
Epoch 1/3000
86028/86028 [==============================] - 16s 185us/step - loss: -0.0560 - acc: 8.8343e-04 - val_loss: -0.0616 - val_acc: 1.0989e-04

Loss is negative, why?

On the visualization profit section:
Since 2017.06.26 up to 2017.06.28, the positions are 1s but no profit at all. It is not working properly right?

Hi @happynoom

I wonder how do you come up with this specific kind of model structure or skeleton. Could you explain the rationality for designing the model in such a structure?

Also, you have chose 61 indicators and normalized them in certain ways. Could you explain why you choose those indicators and why you normalize them in such a way?

Thanks

我看到chart.py里有一部分数据，使用到np.mean(),例如norm_volumes = (volumes - numpy.mean(volumes)) / math.sqrt(numpy.var(volumes))，这样训练及测试集一起处理的时候，会用到未来数据吧。如果不一起处理的话，这个mean会很不稳定。

1. Do you train a single model for you whole data set, or one per asset?
2. How often do you retrain the model?

That sounds simple,but sorry,I have to admit it really puzzles me.