happynoom / deeptrade_keras Goto Github PK
View Code? Open in Web Editor NEWHome Page: http://www.deeplearning.xin
Home Page: http://www.deeplearning.xin
when running it, there is an error saying can't find the file names with ultimate_feature
, ultimate_label
and etc.
Here is the error:
File "/Users/Natsume/miniconda2/envs/tf-experiment/lib/python3.6/site-packages/numpy/lib/npyio.py", line 860, in loadtxt
fh = iter(open(fname))
FileNotFoundError: [Errno 2] No such file or directory: './ultimate_feature.30'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
The source code involved:
23 def read_ultimate(path, input_shape):
24 -> ultimate_features = numpy.loadtxt(path + "ultimate_feature." + str(input_sha
25 ultimate_features = numpy.reshape(ultimate_features, [-1, input_shape[0], in
26 ultimate_labels = numpy.loadtxt(path + "ultimate_label." + str(input_shape[0
27 # ultimate_labels = numpy.reshape(ultimate_labels, [-1, 1])
28 train_set = DataSet(ultimate_features, ultimate_labels)
29 test_features = numpy.loadtxt(path + "ultimate_feature.test." + str(input_sh
30 test_features = numpy.reshape(test_features, [-1, input_shape[0], input_shap
31 test_labels = numpy.loadtxt(path + "ultimate_label.test." + str(input_shape[
32 # test_labels = numpy.reshape(test_labels, [-1, 1])
33 test_set = DataSet(test_features, test_labels)
34 return train_set, test_set
by the way, your code is compatible with keras 2, right?
@happynoom why do you choose relu_limited over sigmoid?
It looks like relu_limited cut off things, whereas sigmoid still consider very negative and very large positive numbers.
Have you tried both and found relu_limited outperform sigmoid?
在char文件里面,X特征用户第window天的信息(假设window=30, X包含了第30天和30天以前的的close price),p_change 是第30天和第29天的price变化率。 这个穿越太明显了
x = feature_arr[:, p:p + window]
p_change = self.make_label(close_prices[p + window - 1: p + window + self.prospective])
@happynoom ,首先,谢谢大佬的分享!投资小白,请教几个问题
1,拜读了一下您的代码,您帮我看一下我的理解对不对:关于数据部分看到dataset下,有26个文件,猜测是26支标的(对应代码),1991年至2007年6464(或不到)天的开,最高,收,最低,交易量的数据。您将这些所有数据,按照日期先后划分,得到一个train,val部分。每个部分由这26支标的所提取的特征(根据selector选取特征),得到输入为[30, 73]的输入,30为窗口大小(每次输入网络某一个标的30天的数据),73为每日特征的长度,预测得到一个P(您在README讲的很清楚了,不再赘述)
2,为什么要选取这26支标的呢?(可以认为都是成分股或者股指?),那老师有试验过,加入更多数据么,结果如何呢?有没有考虑过正负样本的问题呢?
3,关于label的设定,我看到是这样设定的,并不是纯粹的变化率,您能解释一下么?
Lines 31 to 39 in 99511c7
谢谢!期待您的回答
Thank you for providing the concept worthy of my reference. But I encountered the problem of visualizing the chart after running the forecast. I would like to ask here, can you provide the code of the website (like http://www.deeplearning.xin)? If you can provide it, I am really grateful.
Hi @happynoom
you have been using test set as validation set in training.
How about making training set, validation set and test set instead of just training set and test set( which is used as both validation and test set)? It should help to see whether model is really robust or not.
Hi @happynoom
Is this model at L25 the best model you tried so far?
Why do you use dropout on the first layer or input_layer? Wouldn't it be more usual to apply dropout on hidden layers?
thanks
python3.5 tensorflow=1.5 keras=2.1,there is error.
MacBook-Pro:DeepTrade_keras boliu$ python gossip.py
/Users/boliu/anaconda3/lib/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float
to np.floating
is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type
.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
initializing..., learing rate 0.001, n_layers 1, n_hidden 16, dropout rate 0.3.
WARNING:tensorflow:From /Users/boliu/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1242: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
2018-07-25 17:04:10.161405: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
Traceback (most recent call last):
File "gossip.py", line 124, in
make_model([30, 61], 3000, 512, lr=0.001)
File "gossip.py", line 92, in make_model
wp = WindPuller(input_shape=input_shape, lr=lr, n_layers=n_layers, n_hidden=n_hidden, rate_dropout=rate_dropout)
File "/Volumes/BOOTCAMP/dlnlp/DeepTrade_keras/windpuller.py", line 46, in init
metrics=['accuracy'])
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/models.py", line 821, in compile
**kwargs)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 666, in compile
loss_function = losses.get(loss)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/losses.py", line 113, in get
return deserialize(identifier)
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/losses.py", line 105, in deserialize
printable_module_name='loss function')
File "/Users/boliu/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 159, in deserialize_keras_object
':' + function_name)
ValueError: Unknown loss function:risk_estimation
MacBook-Pro:DeepTrade_keras boliu$
I print out pred below and values don't look like the price of prediction of next day.
pred = saved_wp.predict(test_set.images, 1024).
How can I get prediction of next day stock price?
Thanks,
Hi,
Thank you for putting this together. But I am unable to find a way train the model with a new data set. How do I need to prepare the ultimate feature and label files.
is there a way to adapt your code to work on Forex with 1 hour timeframe . The CSV datafiles is have columns:
DateTime, Open, High , Low , Close , Volume
or
EpochTime, Open, High , Low , Close , Volume
so that it would output buy or sell signal for next hour.
Hi happynoom,
What was your accuracy when you trained the model. I tried to train it with my dataset and I got a very low accuracy (10-14%). Although with low accuracy, the model performs considerably well.
Hello, nice code, congrats!
I have a question regarding the 2 functions above.
Can relu_limited and risk_estimation be defined inside the python code instead of $PYTHON_DIR/dist-packages/keras/losses.py and $PYTHON_DIR/dist-packages/keras/activations.py ?
I want to run this in a docker container and it would be simpler if I can define them, for instance, in gossip.py.
Please let me know what you think about it.
Hi @happynoom
I have tried the model.30.best
come along with your repo. is it the best model with the weights you trained over 3000 epochs?
Have you tried to plot a cumulative return curve with predictions using the best trained model? How good is the curve? Could you upload the image too?
Thanks
Just run model fitting and I am getting:
Train on 86028 samples, validate on 18200 samples
Epoch 1/3000
86028/86028 [==============================] - 16s 185us/step - loss: -0.0560 - acc: 8.8343e-04 - val_loss: -0.0616 - val_acc: 1.0989e-04
Loss is negative, why?
Hi @happynoom
I wonder how do you come up with this specific kind of model structure or skeleton. Could you explain the rationality for designing the model in such a structure?
Also, you have chose 61 indicators and normalized them in certain ways. Could you explain why you choose those indicators and why you normalize them in such a way?
Thanks
我看到chart.py里有一部分数据,使用到np.mean(),例如norm_volumes = (volumes - numpy.mean(volumes)) / math.sqrt(numpy.var(volumes)),这样训练及测试集一起处理的时候,会用到未来数据吧。如果不一起处理的话,这个mean会很不稳定。
That sounds simple,but sorry,I have to admit it really puzzles me.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.