helsinki-nlp / hbmp Goto Github PK

Hi,

Thanks for sharing your code! I have a question on transfer tasks.

When you train your model on SNLI and Multinli for transfer learning, do you fine-tune word embedding during training?

i just want to cite your paper in my paper,can you send the original image of hbmp architecture to me at [email protected] ? Thranks!

Hi, thanks for your contributions.
I have studied your model and I intend to evaluate its results in another language.
After running the following code, I find a RuntimeError in the last line
how to fix this error? please guide me and explain it?

!/bin/bash /content/drive/My\ Drive/HBMP-master/train_hbmp.sh

The device argument should be set by using torch.device or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The device argument should be set by using torch.device or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The device argument should be set by using torch.device or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:46: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.1 and num_layers=1
"num_layers={}".format(dropout, num_layers))

Config: ['--epochs', '20', '--batch_size', '64', '--corpus', 'scitail', '--encoder_type', 'HBMP', '--activation', 'leakyrelu', '--optimizer', 'adam', '--word_embedding', 'glove.840B.300d', '--embed_dim', '300', '--fc_dim', '600', '--hidden_dim', '600', '--layers', '1', '--dropout', '0.1', '--learning_rate', '0.0005', '--lr_patience', '1', '--lr_decay', '0.99', '--lr_reduction_factor', '0.2', '--weight_decay', '0', '--early_stopping_patience', '3', '--save_path', 'results', '--seed', '1234']

Namespace(activation='leakyrelu', batch_size=64, cells=2, corpus='scitail', dropout=0.1, early_stopping_patience=3, embed_dim=300, embed_size=22693, encoder_type='HBMP', epochs=20, fc_dim=600, gpu=0, hidden_dim=600, layers=1, learning_rate=0.0005, lower=True, lr_decay=0.99, lr_patience=1, lr_reduction_factor=0.2, optimizer='adam', out_dim=2, resume_snapshot='', save_path='results', seed=1234, weight_decay=0.0, word_embedding='glove.840B.300d')
Model:

NLIModel(
(sentence_embedding): SentenceEmbedding(
(word_embedding): Embedding(22693, 300)
(encoder): HBMP(
(max_pool): AdaptiveMaxPool1d(output_size=1)
(rnn1): LSTM(300, 600, dropout=0.1, bidirectional=True)
(rnn2): LSTM(300, 600, dropout=0.1, bidirectional=True)
(rnn3): LSTM(300, 600, dropout=0.1, bidirectional=True)
)
)
(classifier): FCClassifier(
(activation): LeakyReLU(negative_slope=0.01)
(mlp): Sequential(
(0): Dropout(p=0.1)
(1): Linear(in_features=14400, out_features=600, bias=True)
(2): LeakyReLU(negative_slope=0.01)
(3): Dropout(p=0.1)
(4): Linear(in_features=600, out_features=600, bias=True)
(5): LeakyReLU(negative_slope=0.01)
(6): Linear(in_features=600, out_features=2, bias=True)
)
)
)

Parameters: 28799102

Training started...

Epoch: 01/20 (Learning rate: 0.0005)
Traceback (most recent call last):
File "train.py", line 393, in
main()
File "train.py", line 260, in main
answer = model(batch)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/HBMP-master/classifier.py", line 57, in forward
prem = self.sentence_embedding(batch.premise)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/HBMP-master/embeddings.py", line 19, in forward
sentence = self.word_embedding(input_sentence)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 118, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1454, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'

When i want to replay your code , i meeting a problem :
AttributeError: type object 'SciTail' has no attribute 'urls'. @jorgtied

do you have the source code on the other platform，keras ，tf？

Hello, my friend! When I run the model, I found that the version of the dependency which model need is too old , and now we use pytorch 1.8.0+、python 3.7+. So I just want to know this model could run in the newest pytorch version? More important , i find a problem that :

RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /tmp/pip-req-build-ufslq_a9/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /tmp/pip-req-build-ufslq_a9/caffe2/serialize/inline_container.cc:132)

and my environment is:
pytorch(1.8.1)
spacy(3.0.5)
torchtext(0.9.1)
cudatoolkit(cudatoolkit)

What should i do for this problem . Look forward your early reply!

Hello! My friend.
I am trying to reproduce the HBMP model(trained on SNLI) on SentEval tool.But i found my result(83.15%) is always 1.0-1.5% lower than the state of the art 84.6% on CR from Conneau's paper. And the result on MR is the same. other settings on model, code are same with your paper.
Here are some details：
1.results(just compared to Infersent)
Infersent results from anthors'paper: CR(84.6%), MR(79.9%) (on SNLI)
HBMP(on SNLI) reproduced results :
2021-06-09 10:17:48,408 : ***** Transfer task : CR *****
2021-06-09 10:18:03,835 : Generated sentence embeddings
2021-06-09 10:18:03,836 : Training pytorch-MLP-nhid0-adam,lr=0.01-bs64 with (inner) 10-fold cross-validation
2021-06-09 10:21:59,686 : Best param found at split 1: l2reg = 0.001 with score 84.84
2021-06-09 10:24:56,416 : Best param found at split 2: l2reg = 0.01 with score 84.37
2021-06-09 10:27:09,470 : Best param found at split 3: l2reg = 1e-05 with score 84.16
2021-06-09 10:29:28,407 : Best param found at split 4: l2reg = 0.01 with score 84.75
2021-06-09 10:31:51,597 : Best param found at split 5: l2reg = 0.01 with score 84.69
2021-06-09 10:34:45,882 : Best param found at split 6: l2reg = 0.001 with score 84.49
2021-06-09 10:38:23,245 : Best param found at split 7: l2reg = 1e-05 with score 84.58
2021-06-09 10:42:02,684 : Best param found at split 8: l2reg = 0.01 with score 84.17
2021-06-09 10:46:16,299 : Best param found at split 9: l2reg = 0.01 with score 85.02
2021-06-09 10:50:04,303 : Best param found at split 10: l2reg = 1e-05 with score 84.43
2021-06-09 10:50:11,270 : Dev acc : 84.55 Test acc : 83.15

2021-06-09 10:50:11,274 : ***** Transfer task : MR *****
2021-06-09 10:50:40,868 : Generated sentence embeddings
2021-06-09 10:50:40,870 : Training pytorch-MLP-nhid0-adam,lr=0.01-bs64 with (inner) 10-fold cross-validation
2021-06-09 11:00:27,195 : Best param found at split 1: l2reg = 0.001 with score 79.22
2021-06-09 11:07:01,380 : Best param found at split 2: l2reg = 0.001 with score 78.97
2021-06-09 11:13:29,129 : Best param found at split 3: l2reg = 0.01 with score 78.56
2021-06-09 11:20:28,882 : Best param found at split 4: l2reg = 0.01 with score 78.66
2021-06-09 11:27:06,542 : Best param found at split 5: l2reg = 0.01 with score 78.62
2021-06-09 11:33:35,779 : Best param found at split 6: l2reg = 0.001 with score 78.68
2021-06-09 11:39:22,995 : Best param found at split 7: l2reg = 0.01 with score 78.54
2021-06-09 11:45:15,899 : Best param found at split 8: l2reg = 0.001 with score 78.65
2021-06-09 11:52:09,794 : Best param found at split 9: l2reg = 0.001 with score 78.73
2021-06-09 11:59:05,532 : Best param found at split 10: l2reg = 0.001 with score 79.23
2021-06-09 11:59:19,889 : Dev acc : 78.79 Test acc : 77.85

2.My senteval settings:
params_senteval = {'task_path': PATH_TRANSFER_TASKS, 'usepytorch': True, 'kfold': 10}
params_senteval['classifier'] = {'nhid': 0, 'optim': 'adam,lr=0.01', 'batch_size': 64,
'tenacity': 5, 'epoch_size': 4}

How could i do for reproducing the results from paper?
change the senteval settings?

Everytime i runing your code i need re-download the glove.840B.300d.zip：
I run your code many times but everytime ,it will download the glove.840B.300d.zip.
I find lines 154 code : “os.path.isfile(pretrained_embedding)” is always Flase , that mabey path proble