helsinki-nlp / hbmp Goto Github PK
View Code? Open in Web Editor NEWSentence Embeddings in NLI with Iterative Refinement Encoders
License: MIT License
Sentence Embeddings in NLI with Iterative Refinement Encoders
License: MIT License
Hello, my friend! When I run the model, I found that the version of the dependency which model need is too old , and now we use pytorch 1.8.0+、python 3.7+. So I just want to know this model could run in the newest pytorch version? More important , i find a problem that :
RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /tmp/pip-req-build-ufslq_a9/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /tmp/pip-req-build-ufslq_a9/caffe2/serialize/inline_container.cc:132)
and my environment is:
pytorch(1.8.1)
spacy(3.0.5)
torchtext(0.9.1)
cudatoolkit(cudatoolkit)
What should i do for this problem . Look forward your early reply!
do you have the source code on the other platform,keras ,tf?
When i want to replay your code , i meeting a problem :
AttributeError: type object 'SciTail' has no attribute 'urls'. @jorgtied
Hi, thanks for your contributions.
I have studied your model and I intend to evaluate its results in another language.
After running the following code, I find a RuntimeError in the last line
how to fix this error? please guide me and explain it?
!/bin/bash /content/drive/My\ Drive/HBMP-master/train_hbmp.sh
The device
argument should be set by using torch.device
or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The device
argument should be set by using torch.device
or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The device
argument should be set by using torch.device
or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:46: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.1 and num_layers=1
"num_layers={}".format(dropout, num_layers))
Config: ['--epochs', '20', '--batch_size', '64', '--corpus', 'scitail', '--encoder_type', 'HBMP', '--activation', 'leakyrelu', '--optimizer', 'adam', '--word_embedding', 'glove.840B.300d', '--embed_dim', '300', '--fc_dim', '600', '--hidden_dim', '600', '--layers', '1', '--dropout', '0.1', '--learning_rate', '0.0005', '--lr_patience', '1', '--lr_decay', '0.99', '--lr_reduction_factor', '0.2', '--weight_decay', '0', '--early_stopping_patience', '3', '--save_path', 'results', '--seed', '1234']
Namespace(activation='leakyrelu', batch_size=64, cells=2, corpus='scitail', dropout=0.1, early_stopping_patience=3, embed_dim=300, embed_size=22693, encoder_type='HBMP', epochs=20, fc_dim=600, gpu=0, hidden_dim=600, layers=1, learning_rate=0.0005, lower=True, lr_decay=0.99, lr_patience=1, lr_reduction_factor=0.2, optimizer='adam', out_dim=2, resume_snapshot='', save_path='results', seed=1234, weight_decay=0.0, word_embedding='glove.840B.300d')
Model:
NLIModel(
(sentence_embedding): SentenceEmbedding(
(word_embedding): Embedding(22693, 300)
(encoder): HBMP(
(max_pool): AdaptiveMaxPool1d(output_size=1)
(rnn1): LSTM(300, 600, dropout=0.1, bidirectional=True)
(rnn2): LSTM(300, 600, dropout=0.1, bidirectional=True)
(rnn3): LSTM(300, 600, dropout=0.1, bidirectional=True)
)
)
(classifier): FCClassifier(
(activation): LeakyReLU(negative_slope=0.01)
(mlp): Sequential(
(0): Dropout(p=0.1)
(1): Linear(in_features=14400, out_features=600, bias=True)
(2): LeakyReLU(negative_slope=0.01)
(3): Dropout(p=0.1)
(4): Linear(in_features=600, out_features=600, bias=True)
(5): LeakyReLU(negative_slope=0.01)
(6): Linear(in_features=600, out_features=2, bias=True)
)
)
)
Parameters: 28799102
Training started...
Epoch: 01/20 (Learning rate: 0.0005)
Traceback (most recent call last):
File "train.py", line 393, in
main()
File "train.py", line 260, in main
answer = model(batch)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/HBMP-master/classifier.py", line 57, in forward
prem = self.sentence_embedding(batch.premise)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/HBMP-master/embeddings.py", line 19, in forward
sentence = self.word_embedding(input_sentence)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 118, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1454, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
Hello! My friend.
I am trying to reproduce the HBMP model(trained on SNLI) on SentEval tool.But i found my result(83.15%) is always 1.0-1.5% lower than the state of the art 84.6% on CR from Conneau's paper. And the result on MR is the same. other settings on model, code are same with your paper.
Here are some details:
1.results(just compared to Infersent)
Infersent results from anthors'paper: CR(84.6%), MR(79.9%) (on SNLI)
HBMP(on SNLI) reproduced results :
2021-06-09 10:17:48,408 : ***** Transfer task : CR *****
2021-06-09 10:18:03,835 : Generated sentence embeddings
2021-06-09 10:18:03,836 : Training pytorch-MLP-nhid0-adam,lr=0.01-bs64 with (inner) 10-fold cross-validation
2021-06-09 10:21:59,686 : Best param found at split 1: l2reg = 0.001 with score 84.84
2021-06-09 10:24:56,416 : Best param found at split 2: l2reg = 0.01 with score 84.37
2021-06-09 10:27:09,470 : Best param found at split 3: l2reg = 1e-05 with score 84.16
2021-06-09 10:29:28,407 : Best param found at split 4: l2reg = 0.01 with score 84.75
2021-06-09 10:31:51,597 : Best param found at split 5: l2reg = 0.01 with score 84.69
2021-06-09 10:34:45,882 : Best param found at split 6: l2reg = 0.001 with score 84.49
2021-06-09 10:38:23,245 : Best param found at split 7: l2reg = 1e-05 with score 84.58
2021-06-09 10:42:02,684 : Best param found at split 8: l2reg = 0.01 with score 84.17
2021-06-09 10:46:16,299 : Best param found at split 9: l2reg = 0.01 with score 85.02
2021-06-09 10:50:04,303 : Best param found at split 10: l2reg = 1e-05 with score 84.43
2021-06-09 10:50:11,270 : Dev acc : 84.55 Test acc : 83.15
2021-06-09 10:50:11,274 : ***** Transfer task : MR *****
2021-06-09 10:50:40,868 : Generated sentence embeddings
2021-06-09 10:50:40,870 : Training pytorch-MLP-nhid0-adam,lr=0.01-bs64 with (inner) 10-fold cross-validation
2021-06-09 11:00:27,195 : Best param found at split 1: l2reg = 0.001 with score 79.22
2021-06-09 11:07:01,380 : Best param found at split 2: l2reg = 0.001 with score 78.97
2021-06-09 11:13:29,129 : Best param found at split 3: l2reg = 0.01 with score 78.56
2021-06-09 11:20:28,882 : Best param found at split 4: l2reg = 0.01 with score 78.66
2021-06-09 11:27:06,542 : Best param found at split 5: l2reg = 0.01 with score 78.62
2021-06-09 11:33:35,779 : Best param found at split 6: l2reg = 0.001 with score 78.68
2021-06-09 11:39:22,995 : Best param found at split 7: l2reg = 0.01 with score 78.54
2021-06-09 11:45:15,899 : Best param found at split 8: l2reg = 0.001 with score 78.65
2021-06-09 11:52:09,794 : Best param found at split 9: l2reg = 0.001 with score 78.73
2021-06-09 11:59:05,532 : Best param found at split 10: l2reg = 0.001 with score 79.23
2021-06-09 11:59:19,889 : Dev acc : 78.79 Test acc : 77.85
2.My senteval settings:
params_senteval = {'task_path': PATH_TRANSFER_TASKS, 'usepytorch': True, 'kfold': 10}
params_senteval['classifier'] = {'nhid': 0, 'optim': 'adam,lr=0.01', 'batch_size': 64,
'tenacity': 5, 'epoch_size': 4}
How could i do for reproducing the results from paper?
change the senteval settings?
i just want to cite your paper in my paper,can you send the original image of hbmp architecture to me at [email protected] ? Thranks!
Hi,
Thanks for sharing your code! I have a question on transfer tasks.
When you train your model on SNLI and Multinli for transfer learning, do you fine-tune word embedding during training?
Everytime i runing your code i need re-download the glove.840B.300d.zip:
I run your code many times but everytime ,it will download the glove.840B.300d.zip.
I find lines 154 code : “os.path.isfile(pretrained_embedding)” is always Flase , that mabey path proble
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.