Comments (9)
yzhangcs
commented on August 25, 2024
Does it behave similarly on PTB?
from parser.
attardi
commented on August 25, 2024
I only use UD.
On the UD_English_EWT the release branch achieves:
2020-06-14 20:48:00 INFO Epoch 240 / 1000:
2020-06-14 20:48:54 INFO dev: - loss: 0.7806 - UCM: 56.04% LCM: 45.90% UAS: 87.41% LAS: 82.86%
2020-06-14 20:49:06 INFO test: - loss: 0.8076 - UCM: 50.16% LCM: 38.12% UAS: 89.39% LAS: 85.30%
2020-06-14 20:49:07 INFO 0:01:06.209307s elapsed (saved)
with this configuration:
bert_model | bert-base-cased
n_embed | 100
n_char_embed | 50
n_feat_embed | 100
n_bert_layers | 4
while the dev branch achieves:
Epoch 177 / 1000:
train: Loss: 0.2244 UAS: 95.55% LAS: 92.70%
dev: Loss: 0.6326 UAS: 93.16% LAS: 90.24%
test: Loss: 1.1072 UAS: 91.84% LAS: 89.23%
0:01:00.618501s elapsed (saved)
although with a different model and configuration:
bert_model | TurkuNLP/wikibert-base-en-cased
n_embed | 100
n_char_embed | 50
n_feat_embed | 0
n_bert_layers | 0
embed_dropout | 0.33
n_lstm_hidden | 400
n_lstm_layers | 2
from parser.
attardi
commented on August 25, 2024
I am running the same configuration as the dev branch, but it doesn't look promising.
bert | TurkuNLP/wikibert-base-en-cased
n_embed | 100
n_char_embed | 50
n_feat_embed | 0
n_bert_layers | 0
embed_dropout | 0.33
n_lstm_hidden | 400
n_lstm_layers | 2
At epoch 26, release gets:
2020-06-15 13:58:19 INFO Epoch 26 / 1000:
2020-06-15 13:58:54 INFO dev: - loss: 1.0003 - UCM: 46.15% LCM: 36.16% UAS: 82.75% LAS: 76.09%
2020-06-15 13:59:01 INFO test: - loss: 1.0627 - UCM: 36.27% LCM: 25.12% UAS: 84.05% LAS: 77.21%
2020-06-15 13:59:03 INFO 0:00:42.405768s elapsed (saved)
while the dev branch was already at:
Epoch 26 / 1000:
train: Loss: 0.5479 UAS: 89.65% LAS: 84.42%
dev: Loss: 0.5415 UAS: 91.82% LAS: 88.41%
test: Loss: 0.8552 UAS: 91.26% LAS: 88.26%
from parser.
yzhangcs
commented on August 25, 2024
Sorry, the release branch is still in development and some bugs may lurk in the code.
I will do some checks on PTB later.
from parser.
attardi
commented on August 25, 2024
I did an experiment with the dev branch using the new model electra-base-discrimintator and achieved an improvement on UD_English_EWT.
From bert-base-cased:
Epoch 177 / 1000:
train: Loss: 0.2244 UAS: 95.55% LAS: 92.70%
dev: Loss: 0.6326 UAS: 93.16% LAS: 90.24%
test: Loss: 1.1072 UAS: 91.84% LAS: 89.23%
to electra-base-discriminator:
Epoch 128 / 1000:
train: Loss: 0.2552 UAS: 95.10% LAS: 91.90%
dev: Loss: 0.4805 UAS: 94.49% LAS: 91.90%
test: Loss: 1.1904 UAS: 91.73% LAS: 89.20%
from parser.
yzhangcs
commented on August 25, 2024
I also notice a drop in performance on PTB.
Something maybe erroneously modified.
from parser.
yzhangcs
commented on August 25, 2024
Hi @attardi, the bug has been fixed.
It' because I didn't figure out the usage of from_config. To load the model weights, we should use from_pretrained instead of from_config.
from parser.
attardi
commented on August 25, 2024
I also tested the XLNet model on te dev branch, as you suggested. It is less accurate:
Epoch 147 / 1000:
train: Loss: 0.2343 UAS: 95.27% LAS: 92.30%
dev: Loss: 0.5309 UAS: 93.50% LAS: 90.83%
test: Loss: 1.0762 UAS: 91.94% LAS: 89.35%
from parser.
yzhangcs
commented on August 25, 2024
What does less accurate mean @attardi? Is their any exception on dev branch?
from parser.
Related Issues (20)
- 如何下载dep-biaffine-xlmr的checkpoint？ HOT 4
- 'con-crf-roberta-en' won't load HOT 5
- low performance of "dep-biaffine-xlmr"? HOT 4
- How to using checkpoint to predict HOT 1
- 需要您的帮助 HOT 4
- Doc improvement HOT 1
- Necessity to create a new release HOT 6
- Missing AttachJuxtapose pre-trained weights HOT 3
- Partial Parsing HOT 1
- Constituency CRF RoBERTa English not working properly HOT 2
- Constituency Multilinugal model not working properly HOT 1
- Submodule not found HOT 1
- Unable to use "biaffine-dep-roberta-en" HOT 1
- Global variable access Exception in data utils HOT 4
- 训练新模型遇到问题 HOT 7
- Add verbose flag to remove progress_bar HOT 3
- Missing word_embed with "bert" encoder HOT 3
- TypeError: code expected at most 16 arguments, got 19 HOT 2
- Error Loading Multilingual Dependency Parsing Model HOT 2
- 训练数据格式是什么样的呢？
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from parser.