Issue with executing conv_net_train.py about personality-detection HOT 31 OPEN

Actually I do not remember the exact time required for each step of the code. In the default setting, it requires 50 epochs to complete, and you need to wait for it. You can configure the code as I mentioned in the readme to reduce the training time. Besides, using a GPU will reduce the training time significantly. If you need to understand the implementation, the only thing I can refer to is the Theano website and the paper.

from personality-detection.

I tried running conv_net_train_keras.py on GPU google colab it kept crashing because of use of RAM, and using CPU i got this error

batch size: #32 (comment)

from personality-detection.

from personality-detection.

from personality-detection.

from personality-detection.

Yes. I meant you need to wait until the code completes its task successfully. after that history file will be closed.

from personality-detection.

Try this: https://stackoverflow.com/a/55507901/5661543

from personality-detection.

Hmm... That's strange. Are you using my version or the original version?
can you trace the CPU/GPU usage from task manager (windows), htop command (Linux)?
Can you run this python file in a debug mode using pycharm to trace the exact line that makes this issue?

from personality-detection.

Thanks a lot for your response. Actually because of this issue, I setup your version by installing all the respective modules. When I run the conv_net_train_keras.py in pycharm, I got the following error. I have set up the CUDA on my GPU as well for executing this.

2020-03-30 05:09:51.532398: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 512 totalling 1.0KiB
2020-03-30 05:09:51.532769: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 11 Chunks of size 1024 totalling 11.0KiB
2020-03-30 05:09:51.533163: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.3KiB
2020-03-30 05:09:51.533543: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1536 totalling 1.5KiB
2020-03-30 05:09:51.533929: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 1792 totalling 5.3KiB
2020-03-30 05:09:51.534307: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 240128 totalling 469.0KiB
2020-03-30 05:09:51.534705: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 470784 totalling 459.8KiB
2020-03-30 05:09:51.535114: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 480000 totalling 937.5KiB
2020-03-30 05:09:51.535516: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 547328 totalling 1.57MiB
2020-03-30 05:09:51.535916: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 720128 totalling 2.06MiB
2020-03-30 05:09:51.536345: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 960000 totalling 937.5KiB
2020-03-30 05:09:51.536771: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 6.39MiB
2020-03-30 05:09:51.537159: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 1436473600 memory_limit_: 1436473753 available bytes: 153 curr_region_allocation_bytes_: 2872947712
2020-03-30 05:09:51.537847: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 1436473753
InUse: 6698752
MaxInUse: 6699264
NumAllocs: 178
MaxAllocSize: 960000
2020-03-30 05:09:51.538711: W tensorflow/core/common_runtime/bfc_allocator.cc:429] *___________________________________________________________________________________________________
2020-03-30 05:09:51.540175: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Internal: Dst tensor is not initialized.
[[{{node main_input/_8}}]]
Traceback (most recent call last):

Was it because I'm out of memory in the GPU? If so, please guide me to fix this.

Thanks.

from personality-detection.

Thanks a lot for your response

Your welcome :). You can make us happy with giving our project a star :D

Was it because I'm out of memory in the GPU? If so, please guide me to fix this.

Try reducing the batch_size, try with 1 and if it worked, increase it.
you can also mention @saminfatehir for more specific questions about the Keras model.

from personality-detection.

It worked :) Thanks a lot for helping.
But another issue has been occurred and the console log has been extracted as follows.

Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\JetBrains\PyCharm 2019.1.3\plugins\python\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "C:\Program Files\JetBrains\PyCharm 2019.1.3\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "E:/Projects/personality-detection-Keras/conv_net_train_keras.py", line 316, in
results = train_conv_net(datasets, W, historyfile, i)
File "E:/Projects/personality-detection-Keras/conv_net_train_keras.py", line 176, in train_conv_net
callbacks=[my_logger, history])
File "C:\Python37\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Python37\lib\site-packages\keras\engine\training.py", line 1732, in fit_generator
initial_epoch=initial_epoch)
File "C:\Python37\lib\site-packages\keras\engine\training_generator.py", line 260, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Python37\lib\site-packages\keras\callbacks\callbacks.py", line 152, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "E:/Projects/personality-detection-Keras/conv_net_train_keras.py", line 52, in on_epoch_end
logging.info("epoch = %4d loss = %0.6f acc = %0.2f%%" % (epoch, curr_loss, curr_acc))
TypeError: must be real number, not NoneType

Was it because of the Keras version? I'm using the latest Keras version on Tensorflow 2.1.0.

from personality-detection.

Hmm, I think it may be because of the value of (epoch,curr_loss,curr_acc). Are you passing an appropriate dataset containing instances for both classes (0.1 of your training data is your validation dataset) ?
Can you trace in debug mode and specify which attribute is None(before the line of error, there are some evaluation results)?
It also maybe because of the version of keras or TF, but I am not sure at the moment.

from personality-detection.

It worked :) I changed the followings
curr_acc = logs.get('acc')
val_acc = logs.get('val_acc')

to
curr_acc = logs.get('accuracy')
val_acc = logs.get('val_accuracy')

As the next step, how should I test this model? As an example, when a sample sentence has been given as input, how this model work using the trained model? Please explain me the way of converting this model to real time context.

Thank you very much.

from personality-detection.

Actually, it needs some effort. First, you should save the final model instead of testing it on the test data.
Then you should prepare your test set. Note that you should feed the full test dataset to Mairesse and after that, you should run the process-data python file and use its extracted features as the input of the trained model.

from personality-detection.

Could you please elaborate more on this? Thanks a lot.

from personality-detection.

The problem is that this code is designed for testing on the Essays dataset. If you want to feed another data, you have two options:

1. Change the code so that it can read and process your data
2. Change your data so that it can be read by the code

In both of these directions, you need to extract mairesse psycholinguistics features. full instruction is available here. Then you need to feed your data into the model. First, you should save the trained model (I mentioned where you should do that). Second, since the code is designed for 10 fold cross-validation, change it in order to handle your test samples and preprocess it without dividing it into 10 folds. Third, you need to feed the preprocessed data (embeddings and extracted mairesse features) into the saved model to evaluate and predict the label.
Hope this helps.

from personality-detection.

@amirmohammadkz I am trying to run your code but I am having this error

File "/Volumes/SAMSUNG/personality-detection-master-amirmohamed/conv_net_train.py", line 508, in
activations=[Sigmoid])

File "/Volumes/SAMSUNG 1/personality-detection-master-amirmohamed/conv_net_train.py", line 143, in train_conv_net
dropout_rates=dropout_rate)

File "", line 106, in init

TypeError: 'zip' object is not subscriptable

any idea how to fix this

Thanks for using my forked repository. Are you using Python 2.7 ? Please use these requirements.

from personality-detection.

@amirmohammadkz I have one more question

the terminal is stuck on this should the process terminate automatically or do I have to do it ?

can you refer me to any resources to understand the code implementation more (courses or videos)

Thanks in advance and sorry to bother you.

from personality-detection.

but I tried running on mac I got this

the first epoch finished but then this error popped up

Have you tested this solution? #32 (comment)

from personality-detection.

but I tried running on mac I got this
the first epoch finished but then this error popped up

Have you tested this solution? #32 (comment)

I have already trained the model, but after training what should i do? is this project meant only for training or is there is any code for testing the trained model. conv_net_classes.py:- What is the use of this python file.

from personality-detection.

but I tried running on mac I got this
the first epoch finished but then this error popped up

Have you tested this solution? #32 (comment)

I have already trained the model, but after training what should i do? is this project meant only for training or is there is any code for testing the trained model. conv_net_classes.py:- What is the use of this python file.

It will automatically test the trained model on the dafult test set. you can change the test set if you want.#32 (comment)

from personality-detection.

but I tried running on mac I got this
the first epoch finished but then this error popped up

Have you tested this solution? #32 (comment)

I have already trained the model, but after training what should i do? is this project meant only for training or is there is any code for testing the trained model. conv_net_classes.py:- What is the use of this python file.

It will automatically test the trained model on the dafult test set. you can change the test set if you want.#32 (comment)

then what is the use of conv_net_classes.py

from personality-detection.

As far as I can remember, it is about the implementation of layers.

from personality-detection.

Now its training smoothly isnt it. If not, yes reduce the epochs as well.
…

does this message mean the the output is correct ?
"Buffered data was truncated after reaching the output size limit."

also "history_main_model_attr_0_w2v.txt" and "perf_output_main_model_0_w2v.txt" files are empty, the only file have data is the log file.

@CyraxSector and thanks for your response

That's because the file is already open. In logging scenario, the code manages the errors and allows you to monitor the logs at any time, but for the result, result files should be closed at the end of the training in order to be available.

from personality-detection.

@amirmohammadkz I've tested the model successfully by following all your steps. As an example, when a sample sentence has been given as input, it will be evaluated. Thanks a lot for all your guidance. I'll make the PR ASAP.

from personality-detection.

@amirmohammadkz I've tested the model successfully by following all your steps. As an example, when a sample sentence has been given as input, it will be evaluated. Thanks a lot for all your guidance. I'll make the PR ASAP.

How you have tested the model. As described in the comments training and testing will be done by one code.

from personality-detection.

@amirmohammadkz I've tested the model successfully by following all your steps. As an example, when a sample sentence has been given as input, it will be evaluated. Thanks a lot for all your guidance. I'll make the PR ASAP.

Happy to hear that! Thanks a lot!

from personality-detection.

Hello,
If you are looking for a faster version of personality detection, our new paper, and its code is available here:
https://github.com/amirmohammadkz/personality_detection

This code is significantly faster and can be run on a regular computer/laptop. The accuracies are higher too.

from personality-detection.

@amirmohammadkz

I am getting this error when I was trying to load 'GoogleNews-vectors-negative300-SLIM.bin'. (Code given below)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-b567896a8aa6> in <module>
     14 print('vocab size: ' + str(len(vocab)))
     15 print('max sentence length: ' + str(max_l))
---> 16 w2v = load_bin_vec(wv_from_bin, vocab)
     17 print(w2v)
     18 print('word2vec loaded!')

<ipython-input-20-59822c213c28> in load_bin_vec(fname, vocab)
     49     """
     50     word_vecs = {}
---> 51     with open(fname, 'rb') as f:
     52         header = f.readline()
     53         vocab_size, layer1_size = map(int, header.split())

TypeError: expected str, bytes or os.PathLike object, not Word2VecKeyedVectors

Code is

w2v_file = 'GoogleNews-vectors-negative300-SLIM.bin'
revs, vocab = build_data_train_test(data_train, train_ratio=0.6, clean_string=True)
max_l = np.max(pd.DataFrame(revs)['num_words'])
print('data loaded!')
print('number of sentences: ' + str(len(revs)))
print('vocab size: ' + str(len(vocab)))
print('max sentence length: ' + str(max_l))
w2v = load_bin_vec(w2v_file, vocab)
print(w2v)
print('word2vec loaded!')
print('num words already in word2vec: ' + str(len(w2v)))

add_unknown_words(w2v, vocab)
W, word_idx_map = get_W(w2v)
cPickle.dump([revs, W, word_idx_map, vocab], open('imdb-train-val-testN.pickle', 'wb'))
print('dataset created successfully!')

Any help or guidance is highly appreciated.

from personality-detection.

@addy1997 Have a look at my forked repo. I think it will be a better starting point.
https://github.com/amirmohammadkz/personality-detection

from personality-detection.

@addy1997 Have a look at my forked repo. I think it will be a better starting point.
https://github.com/amirmohammadkz/personality-detection

Thanks it worked

from personality-detection.