Comments (8)
can you maybe just re-run the binarizing step very quickly and see if you still have the same error? Like, remove the .pth and rerun preprocess.py
If this still does not work, you can send me the .pth of the test set which is small, and I can try to reload it.
from unsupervisedmt.
I did as you suggested and preprocessed files again. Now it seems that newly generated .pth
files work.
I checked out what was written in old and new log files, and in both cases summary infos (# words etc.) were the same.
So I checked the size of old and new .pth
files and it appears that there is a difference - all older .pth
files have 3 bytes more ... I will close this issue, but it is rather interesting why old .pth
files were corrupted.
from unsupervisedmt.
Hi,
This is weird, I'm not aware of such package. What is the full command you used to run the model?
from unsupervisedmt.
Hi,
given that I run the command from NMT directory, the command looks as follows:
python3 main.py --exp_name transformer-deen --transformer True --n_enc_layers 4 --n_dec_layers 4 --share_enc 3 --share_dec 3 --share_lang_emb True --share_output_emb True --langs 'de,en' --n_mono -1 --mono_dataset 'de:../../data/training.de.tok.50000.pth,,;en:../../data/training.en.tok.50000.pth,,' --para_dataset 'de-en:,../../data/dev.tok.XX.50000.pth,../../data/test.tok.XX.50000.pth' --mono_directions 'de,en' --word_shuffle 3 --word_dropout 0.1 --word_blank 0.2 --pivo_directions 'de-en-de,en-de-en' --pretrained_emb '../../data/training.de-en.50000.vec' --pretrained_out True --lambda_xe_mono '0:1,100000:0.1,300000:0' --lambda_xe_otfd 1 --otf_num_processes 30 --otf_sync_params_every 1000 --enc_optimizer adam,lr=0.0001 --epoch_size 500000 --stopping_criterion bleu_de_en_valid,10
from unsupervisedmt.
This looks right. Did you make any changes to the code of the original repository?
Also, do you get the same error when reloading any of the binarized files?
from unsupervisedmt.
I didn't introduce any changes to the original code (PyTorch model). I only changed get_data.sh
script to download and preprocess my datasets, but it seemed to work fine because it generated *.pth files as well as one binary file training.de-en.50000.bin
Also, do you get the same error when reloading any of the binarized files?
How would you like me to test it?
from unsupervisedmt.
You can try to open python in the main directory, import torch
, and then try torch.load(path)
where path
is the path
of the different .pth
files.
from unsupervisedmt.
I tried and it gives the same error message.
To compare if it happens with other serialized models, I loaded some .pt
file from my disk and it didn't return error.
So I guess it seems that .pth
files generated during preprocessing part are "corrupted"? If so, how should I investigate the problem?
Thanks for your help.
from unsupervisedmt.
Related Issues (20)
- why MemoryError
- Why codes file is empty.? HOT 4
- for different language, where to make change?
- How to train NMT + PBSMT ?
- UnboundLocalError: local variable 'n_words' referenced before assignment
- About number of shared layers
- RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [14, 32, 1536]], which is output 0 of AddBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). HOT 1
- How to run PBSMT +NMT ?
- transformer multihead attention scaling layer error
- Setting the random seed does not result in same outputs across runs
- I have trouble when run get_data_enfr.sh
- How can I modify the code to train may own dataset on specific language?
- Low utilization rate of cuda HOT 1
- How to train the vector of phrases
- Low BLEU on PBSMT HOT 3
- bpe_end issue
- Getting raise EOFError() while executing Linux Command through Netmiko
- How i can run MUSE alignment in .sh
- How to train the model without para_dataset
- Error in runny bash command. HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unsupervisedmt.