cderinbogaz / inpredo Goto Github PK

With Kreas 2.2.4 saving checkpoint in this form:

checkpoint = ModelCheckpoint(target_dir, monitor='val_acc', verbose=1, save_best_only=True, mode='max')

is not working. It's giving error:

Traceback (most recent call last):
  File "c:\Users\pjasz\source\repos\inpredo-master\src\train-binary.py", line 76, in <module>
    model.compile(loss='categorical_crossentropy',
  File "C:\Users\pjasz\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\engine\training.py", line 451, in compile
    handle_metrics(output_metrics)
  File "C:\Users\pjasz\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\engine\training.py", line 408, in handle_metrics
    metric_fn = metrics_module.get(metric)
  File "C:\Users\pjasz\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\metrics.py", line 78, in get
    return deserialize(str(identifier))
  File "C:\Users\pjasz\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\metrics.py", line 67, in deserialize
    return deserialize_keras_object(config,
  File "C:\Users\pjasz\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\utils\generic_utils.py", line 164, in deserialize_keras_object
    raise ValueError('Unknown ' + printable_module_name +
ValueError: Unknown metric function:val_acc

Shouldn't we use acc instead?

metric = 'acc'
model.compile(loss='categorical_crossentropy',
                      optimizer=optimizers.rmsprop(),
                      metrics=[metric])
checkpoint = ModelCheckpoint(filepath=target_dir + 'weights-improvement-{epoch:02d}-{acc:.2f}.hdf5', monitor=metric,
                    verbose=2, save_best_only=True, mode='max')

When I run the script for predict-binary.py

I get the following
2019-12-18 16:19:48.785719: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-12-18 16:19:48.801218: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc8d0371a90 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-12-18 16:19:48.801253: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version

but nothing else...
How do you use this script to predict based on the initial training?
Can you please give me an example.

Solutions in this thread:
Title issue
Using a specific model from a checkpoint, not the initial base model (important)
Using the prediction function
Pointing the prediction batch mode to test validation data
Copying random files to validation

Original post:

The training script is crashing out early on with the following output:

`Found 7372 images belonging to 2 classes.
Found 0 images belonging to 0 classes.
Epoch 1/1000
103/103 [==============================] - 266s 3s/step - loss: 210.3966 - accuracy: 0.7679

Traceback (most recent call last):
File "train-binary.py", line 118, in
validation_steps=nb_validation_samples//batch_size)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/keras/engine/training.py", line 1732, in fit_generator
initial_epoch=initial_epoch)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/keras/engine/training_generator.py", line 260, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/keras/callbacks/callbacks.py", line 152, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/keras/callbacks/callbacks.py", line 702, in on_epoch_end
filepath = self.filepath.format(epoch=epoch + 1, **logs)
KeyError: 'val_acc'`

I'm not familiar enough with python to debug this properly and would appreciate any suggestions you might have

thanks!

Is there a way to load a model to continue training? I tried throwing in load_model() right before training starts but it doesn't seem to work.

After Creating Data, It is showing there are 3 classes instead of 2 when I run train-binary.py

Found 2056 images belonging to 3 classes.
Found 448 images belonging to 3 classes.

I'm sure i'm doing something wrong, but I can't figure out what to do next.

I've completed all of the steps up until predict-binary.

I run the script but it prints Using plaidml.keras.backend backend in terminal and then just stops

heres my code, what am I missing? I think i'm missing an image file to predict? but what image file? one of the images graphwerk produced or another?

`import os
import numpy as np
os.environ["KERAS_BACKEND"] = "plaidml.keras.backend"
#os.environ['TF_CPP_MIN_LOG_LEVEL']='2'

from keras.preprocessing.image import ImageDataGenerator, load_img, img_to_array
from keras.models import Sequential, load_model

img_width, img_height = 150, 150
model_path = r'G:\inpredo-master\inpredo\models\model.h5'
weights_path = r'G:\inpredo-master\inpredo\models\weights.h5'
model = load_model(model_path)
test_path = r'G:\inpredo-master\inpredo\data\validation'

def predict(file):
x = load_img(file, target_size=(img_width,img_height))
x = img_to_array(x)
x = np.expand_dims(x, axis=0)
array = model.predict(x)
result = array[0]
if result[0] > result[1]:
if result[0] > 0.9:
print("Predicted answer: Buy")
answer = 'buy'
print(result)
print(array)
else:
print("Predicted answer: Not confident")
answer = 'n/a'
print(result)
return answer
else:
if result[1] > 0.9:
print("Predicted answer: Sell")
answer = 'sell'
print(result)
else:
print("Predicted answer: Not confident")
answer = 'n/a'
print(result)
return answer

def predict_files(test_path):
truebuy = 0
falsebuy = 0
truesell = 0
falsesell = 0
na = 0
for i, ret in enumerate(os.walk(str(test_path) + '/buy')):
for i, filename in enumerate(ret[2]):
if filename.startswith("."):
continue
print("Label: buy")
result = predict(ret[0] + '/' + filename)
if result == "buy":
truebuy += 1
elif result == 'n/a':
print('no action')
na +=1
elif result == 'sell':
falsebuy += 1

for i, ret in enumerate(os.walk(str(test_path) + '/sell')):
for i, filename in enumerate(ret[2]):
if filename.startswith("."):
continue
print("Label: sell")
result = predict(ret[0] + '/' + filename)
if result == "sell":
truesell += 1
elif result == 'n/a':
print('no action')
na += 1
elif result == 'buy':
falsesell += 1

print("True buy: ", truebuy)
print("True sell: ", truesell)
print("False buy: ", falsebuy) # important
print("False sell: ", falsesell)
print("no action:", na)
precision = truesell / (truesell + falsesell)
precision2 = truebuy / (truebuy + falsebuy)
recall = truebuy / (truebuy + falsesell)
print("Sell Precision: ", precision)
print("Buy Precision", precision2)
print("Recall: ", recall)
precision1 = (truesell + truebuy) / (truesell + truebuy + falsesell + falsebuy)
print("Precision: ", precision1)
f_measure = (2 * recall * precision) / (recall + precision)
print("F-measure: ", f_measure)
return precision1, na
`

Hello,

thank you for this interesting repository.
I have run your code as it is but I have noticed that:

• bar colour is inverted (it is red when price descend and green when it increase)
• not every signal is correct, please look at the attached image: it is classified as a sell signal
  

Maybe I am mistaking, could you confirm it is supposed to work like that?

Anyone else here struggling with a very low accuracy? I cant seem to get it above 51% so its about as good as a cointoss. I have an equal amount of buy and sell in my training set as well as validation set.

Hello,
thanks for your Work.
I I launched in Colab according to the instructions. But it gives an error on Second Step.
AttributeError:module 'Tensorflow. _api.v2.config' has no attribute 'experimental_list_devices'
Can you fix it? Or tell me what needs to be done to fix this.

Regards,
Alex

Hi,
when i tried to make buy and sell images, i checked conditional to sell and buy with last date is newer than next date. what is confuse there?

Hey there,

because rows from the .csv file are ordered in the opposite direction, we are actually predicting the past. This is caused by the following operation:

pd = np.flipud(ad)

Given the fact that the close from the current day is the open for the next day, the color of the last bar is actually showing the price for the next (last) day.

Best,
Chris

KeyError: 'accuracy'

Here is the code
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.RMSprop(),
metrics=['accuracy'])
"""""rest of code not shown here. """""""""""""""
"""
Tensorboard log
"""
target_dir = "../models/weights-improvement-{epoch:02d}-{accuracy:.2f}.hdf5"
if not os.path.exists(target_dir):
os.mkdir(target_dir)
model.save('../models/model.h5')
model.save_weights('../models/weights.h5')

checkpoint = ModelCheckpoint(target_dir, monitor='accuracy', verbose=1, save_best_only=True, mode='max')

callbacks_list = [checkpoint]

model.fit_generator(
train_generator,
steps_per_epoch=nb_train_samples//batch_size,
epochs=epochs,
shuffle=True,
validation_data=validation_generator,
callbacks=callbacks_list,
validation_steps=nb_validation_samples//batch_size)

There is an error with the checkpoint as it does not recognize the key 'accuracy'. It is the same key used in compile of the model above as well.

What could be the issue her ?

Thanks for sharing knowledge, i have a question i got that result : Only buy

What shall i do?

Predicted answer: Buy

[1.0000000e+00 1.7195972e-25]

[[1.0000000e+00 1.7195972e-25]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 4.2478185e-26]

[[1.0000000e+00 4.2478185e-26]]

Label: sell

Predicted answer: Buy

[9.9989498e-01 1.0502239e-04]

[[9.9989498e-01 1.0502239e-04]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 1.8285746e-11]

[[1.0000000e+00 1.8285746e-11]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 1.5805315e-20]

[[1.0000000e+00 1.5805315e-20]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 1.3262541e-18]

[[1.0000000e+00 1.3262541e-18]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 2.2895107e-16]

[[1.0000000e+00 2.2895107e-16]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 7.0590938e-12]

[[1.0000000e+00 7.0590938e-12]]

Label: sell

Predicted answer: Buy

[1.000000e+00 8.698008e-14]

[[1.000000e+00 8.698008e-14]]

Label: sell

Predicted answer: Buy

[1.0000000e+00 1.8061724e-14]

[[1.0000000e+00 1.8061724e-14]]

True buy: 111

True sell: 0

False buy: 0

False sell: 122

no action: 0

Sell Precision: 0.0

Buy Precision 1.0

Recall: 0.47639484978540775

Precision: 0.47639484978540775

F-measure: 0.0

Using Google Colab.
python graphwerk.py returns:

60
Traceback (most recent call last):
  File "/content/inpredo/src/graphwerk.py", line 90, in <module>
    graphwerk(iter, iter+12)
  File "/content/inpredo/src/graphwerk.py", line 31, in graphwerk
    open.append(float(pd[start][1]))
ValueError: could not convert string to float: 'BTCUSD'

I edited the ad variable in graphwerk.py to equal

ad = genfromtxt('/content/inpredo/financial_data/btcusd-1h.csv', delimiter=',' ,dtype=str)

Edit: After deleting the second column in btcusd-1h.csv with the string BTCUSD and editing graphwerk.py to specify my buy and sell folders, I now get this error:

60
1.0005000396856893
close value is smaller
last value: 3779.7
next value: 3781.59
buy
0.9955536014781199
close value is bigger
last value: 3821.07
next value: 3804.08
sell


. . . 


Traceback (most recent call last):
  File "/content/inpredo/src/graphwerk.py", line 90, in <module>
    graphwerk(iter, iter+12)
  File "/content/inpredo/src/graphwerk.py", line 39, in graphwerk
    close_next = float(pd[finish][4])
IndexError: index 242 is out of bounds for axis 0 with size 242

Edit: Oh, I see. I overlooked the comment on line 29 in graphwerk.py.

Why author is using

> graphwerk.py


pd = np.flipud(ad)

doesn't it generate horizontally flipped images?
Will this not produce a wrong prediction model?

This is a problem because of the python enviroment you are using. Change the target directory to the exact location instead of a relative path as in the example. For example: "YOUR_PATH_TO_MODELS_FOLDER/models/weights-improvement-{epoch:02d}-{val_acc:2f}.hdf5"

Just go the the folder and check for properties by right clicking and copy the path you see over there to this line.

Originally posted by @cderinbogaz in #11 (comment)

Hello, I added my directory as you specified and still receive the same error

Hi,

Having an issue getting train-binary.py to run correctly on windows due to the colons in the weight-improvement section.
This section : target_dir = "./models/weights-improvement-{epoch:02d}-{val_acc:2f}.hdf5"

Any idea how to get around this?

on the website which makes reference to this repo ('https://towardsdatascience.com/making-a-i-that-looks-into-trade-charts-62e7d51edcba')

function "convolve_sma" is called but not defined
mpl_finance cannot call candlestick2_ochl anymore, mplfinance.original_flavor.candlestick2_ochl seems to work
call time.time() but time isnt imported
realtime_data variable is called but never referenced

you could eliminate like 12 or more lines of code and make things a bit easier to understand with a similar file naming convention

broken tutorials are a huge source of frustration mostly on the new programmers.
every part of this tutorial has similar problems
surely the crypto affiliate is enough incentive to maintain your article/repo, no?

Hey,I am looking to take this project on the level it deserves for our personal growth

Can we speak privately please!