yg211 / bert_nli Goto Github PK
View Code? Open in Web Editor NEWA Natural Language Inference (NLI) model based on Transformers (BERT and ALBERT)
A Natural Language Inference (NLI) model based on Transformers (BERT and ALBERT)
In this code, you have provided .dict
files for BERT_base, BERT_large, and albert_large. I want to download the .dict
file for SciBERT. How can I download that file?
你好,我大致看了一下您的代码,您的方法是,取的pooled_out向量, warm_up + Adamw ,模型参数只有一个线性层,在bert上微调,能够达到约86%的准确率。
我有点关于训练细节的东西想请教一下,取pooled_out 效果会比取-2层好吗,我取-2层,Adam优化器,固定bert参数,只有一个线性层,学习率2e-5,基本只能达到70准确率,我试了微调bert结果基本没有变化,是因为没有warm up 的原因吗,还是说Adamw优化器的原因,您对于这类调参有什么建议吗?
We are running the code on a different dataset than yours and are getting this error.
[...]
evaluate: 100% 723/724 [02:51<00:00, 5.96it/s]
evaluate: 100% 724/724 [02:52<00:00, 4.20it/s]
All predcit: 7176
All labels: 7237
training: 9% 1999/22916 [11:37<2:01:35, 2.87it/s]
Traceback (most recent call last):
File "./train.py", line 195, in <module>
model_dic = train(model, optimizer, scheduler, train_data, dev_data, batch_size, fp16, checkpoint, gpu, max_grad_norm, best_acc)
File "./train.py", line 90, in train
acc = evaluate(model,dev_data,checkpoint,mute=True)
File "/content/drive/MyDrive/bert_nli-master/test_trained_model.py", line 25, in evaluate
assert len(all_predict) == len(all_labels)
AssertionError
We have not changed anything else. Printing the two values all_predict and all_labels actually gives us different lengths.
We don't understand why, could you help us?
there is no output_hidden_states attribute for module
Sorry to distrupt you.
In my attempt, I found that "transformers" has been upgraded, and in that case the "warmup" function cannot be used. Maybe you could consider renew the code in train_test.py.
For example, WarmupLinearSchedule → get_linear_schedule_with_warmup.
The question is that when I used your datasets and tried to train a new model. An error has been raised. It's in utils.py Line 14, tokenized_text[i] is out of range.
torch==1.0.1.post2 in requirements.txt not work for this experiment, I use torch==1.5.0 instead as you suggest in the readme and it works.
Thanks for your work~
I was trying to run the example script, but it seems to give this error Missing key(s) in state_dict: "bert.embeddings.position_ids".
I tried instantiating model = BertNLIModel('output/bert-base.state_dict') but it gives the same error.
Any idea why?
Hey I am finetuning bert base on custom nli dataset. But while starting a training process it is getting stuck at the error stating that torch.nn.modules.module.ModuleAttributeError: 'BertEncoder' object has no attribute 'output_hidden_states'. Can you solve this as as I want to get this done as soon as possible?
2020-12-11 19:57:14 - Lock 140474735309264 acquired on /Users/mac/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157.lock
2020-12-11 19:57:14 - https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin not found in cache or force_download set to True, downloading to /Users/mac/.cache/torch/transformers/tmp_34x9lwo
Downloading: 100%|██████████| 440M/440M [3:20:47<00:00, 36.6kB/s]
Traceback (most recent call last):
File "/Users/mac/Downloads/bert_nli-master/train.py", line 181, in
model = BertNLIModel(gpu=gpu, batch_size=batch_size, bert_type=bert_type, model_path=trained_model, reinit_num=reinit_layers, freeze_layers=freeze_layers)
2020-12-11 23:18:04 - storing https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin in cache at /Users/mac/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
File "/Users/mac/Downloads/bert_nli-master/bert_nli.py", line 21, in init
self.bert = BertModel.from_pretrained('bert-base-uncased')
2020-12-11 23:18:04 - creating metadata file for /Users/mac/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
File "/Users/mac/miniconda3/envs/test_py3/lib/python3.6/site-packages/transformers/modeling_utils.py", line 536, in from_pretrained
model = load_tf2_checkpoint_in_pytorch_model(model, resolved_archive_file, allow_missing_keys=True)
2020-12-11 23:18:04 - Lock 140474735309264 released on /Users/mac/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157.lock
File "/Users/mac/miniconda3/envs/test_py3/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 227, in load_tf2_checkpoint_in_pytorch_model
tf_model = tf_model_class(pt_model.config)
NameError: name 'tf_model_class' is not defined
2020-12-11 23:18:04 - loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at /Users/mac/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
2020-12-11 23:18:05 - Loading TensorFlow weights from /Users/mac/.cache/torch/transformers/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
Process finished with exit code 1
I wanna know how can I solve this problem? I have been downloaded the pre-trained model,but name tf_model_class is not defined.Please help me!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.