Comments (2)
oi02lyl
commented on July 20, 2024
For this dataset, we use the following setting (the summarization option is off). We used roberta, the drop_col da operator, the general dk, and a batch size of 32 for this dataset. The command should look like this:
CUDA_VISIBLE_DEVICES=0 python train_ditto.py \
--task Structured/Beer \
--batch_size 32 \
--max_len 256 \
--lr 3e-5 \
--n_epochs 40 \
--finetuning \
--lm roberta \
--fp16 \
--da drop_col \
--dk general`
I just re-ran the experiment and here are the results:
Baseline (no dk or da)
=========eval at epoch=40=========
Validation:
=============Structured/Beer==================
accuracy=0.967
precision=0.867
recall=0.929
f1=0.897
======================================
Test:
=============Structured/Beer==================
accuracy=0.967
precision=0.824
recall=1.000
f1=0.903
======================================
With DK (general) only
Somehow the test result is the same as the baseline.
=========eval at epoch=40=========
Validation:
=============Structured/Beer==================
accuracy=0.967
precision=0.824
recall=1.000
f1=0.903
======================================
Test:
=============Structured/Beer==================
accuracy=0.967
precision=0.824
recall=1.000
f1=0.903
======================================
Ditto (with both dk and da)
=========eval at epoch=40=========
Validation:
=============Structured/Beer==================
accuracy=0.956
precision=0.857
recall=0.857
f1=0.857
======================================
Test:
=============Structured/Beer==================
accuracy=0.978
precision=0.875
recall=1.000
f1=0.933
======================================
In the paper's experiments, we reported the results by taking the epoch with the highest validation F1 score (here I am taking the last epoch).
I did notice something strange as the baseline result seems significantly higher than before. I remember that we updated the MixDA code from snippext which might have some effect on the data augmentation results too.
from ditto.
CristhianBoujon
commented on July 20, 2024
Thank you!
from ditto.
Related Issues (20)
- Code implementation - Training HOT 9
- ImportError: cannot import name 'LongformerModel' from 'transformers' (transformers=2.8)
- How is your F1 score calculated, whether you use weight or macro or micro HOT 1
- random result when inference 2 similar textual data HOT 3
- Add --save_model flag to the training example
- Error when using --summarize with matcher.py HOT 1
- Whether are special tokens like [COL] [VAL] and attribute names added into the vocabulary?
- [Question] Can I use this package in a notebook environment?
- Inferencing HOT 1
- How (code) to serialize the inputs ? HOT 1
- ModuleNotFoundError: No module named 'click._bashcomplete' HOT 1
- ValueError: not enough values to unpack (expected 2, got 1) - Textual/Company HOT 5
- drop_col gives error? HOT 5
- Which f1 should we report? HOT 1
- evaluation method seems to assign a new f1 value as a best score without computing f1 value by best_th=0.5 HOT 1
- Adding custom tokens
- F1 Score for Structured/Beer on paper can't be reproduced
- The link for the Company.zip file seems to be invalid. HOT 1
- Summarization sometimes removes attribute names between [COL] and [VAL]
- training HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ditto.