weijia-xu / fairseq-editor Goto Github PK
View Code? Open in Web Editor NEWEDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints
License: MIT License
EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints
License: MIT License
can you share your checkpoint (WMT14 ende ...) that trained with 300k update times
I follow the instruction from readme, but can't re-implement your WMT14 en-de result 26.9 BLEU. I got 16.49 BLEU, and only differences are that the max-tokens I set 8000 and I trained the model on two 3090 GPUs with update-freq 4 to imitate 8 GPUs settings
pip
, source):Hi, thanks for sharing the coding! I want to implement the EDITOR using English-Chinese datasets. And I have some question:
pip
, source):Is there any way to fine-tune the model with my data by first loading the model with some pretrained weights? Is there any way to load the model with auto-regressive transformer's pretrained weights? I want to try EDITOR on English-Hindi corpus and I don't have this much of GPUs available with me.
pip
, source):I tried to train a model from English to Chinese, but there was a mistake in the training.
Traceback (most recent call last): File "/share/qzz/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args) File "/share/qzz/en2zh/fairseq-editor/fairseq_cli/train.py", line 286, in distributed_main main(args, init_distributed=True) File "/share/qzz/en2zh/fairseq-editor/fairseq_cli/train.py", line 96, in main train(args, trainer, task, epoch_itr) File "/share/qzz/anaconda3/lib/python3.7/contextlib.py", line 74, in inner return func(*args, **kwds) File "/share/qzz/en2zh/fairseq-editor/fairseq_cli/train.py", line 176, in train log_output = trainer.train_step(samples) File "/share/qzz/anaconda3/lib/python3.7/contextlib.py", line 74, in inner return func(*args, **kwds) File "/share/qzz/en2zh/fairseq-editor/fairseq/trainer.py", line 319, in train_step ignore_grad=is_dummy_batch, File "/share/qzz/en2zh/fairseq-editor/fairseq/tasks/translation_lev.py", line 175, in train_step loss, sample_size, logging_output = criterion(model, sample) File "/share/qzz/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/share/qzz/en2zh/fairseq-editor/fairseq/criterions/nat_loss.py", line 218, in forward factor=outputs[obj].get("factor", 1.0) File "/share/qzz/en2zh/fairseq-editor/fairseq/criterions/nat_loss.py", line 53, in _compute_loss outputs, targets = outputs[masks], targets[masks] IndexError: The shape of the mask [56, 32] at index 1 does not match the shape of the indexed tensor [56, 33] at index 1
I use the parameters:
fairseq-train \ ${data_dir}/data-bin \ --save-dir ${model_dir}/checkpoints \ --source-lang ${src} --target-lang ${tgt} \ --ddp-backend=no_c10d \ --task translation_lev \ --criterion nat_loss \ --arch editor_transformer \ --noise random_delete \ --optimizer adam --adam-betas '(0.9,0.98)' \ --lr 0.001 --lr-scheduler inverse_sqrt \ --warmup-updates 2000 \ --warmup-init-lr '1e-07' --label-smoothing 0.1 \ --share-all-embeddings --no-share-discriminator \ --dropout 0.3 --weight-decay 0.01 \ --decoder-learned-pos --encoder-learned-pos \ --apply-bert-init \ --log-format 'simple' \ --log-interval 5 \ --fixed-validation-seed 7 \ --max-tokens 2048 \ --save-interval-updates 10000 \ --max-update 100000
I tried to change arch to levenshtein_transformer, and it can train normally.
pip
, source):I want to reproduce reported results of the EDITOR model in the main article (table2) for ro-en dataset.
How long does it take to train that ? How much memory is required? Can I reproduce results of EDITOR by google colab free gpu and also train new models?
What's your environment?
fairseq Version (e.g., 1.0 or master):
PyTorch Version (e.g., 1.0):
OS (e.g., Linux): 18.04
How you installed fairseq (pip, source):
Python version:
CUDA/cuDNN version: 11.4
GPU models and configuration: two NVIDIA 1080
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.