bigscience-workshop / multilingual-modeling Goto Github PK
View Code? Open in Web Editor NEWBLOOM+1: Adapting BLOOM model to support a new unseen language
Home Page: https://arxiv.org/abs/2212.09535
License: Apache License 2.0
BLOOM+1: Adapting BLOOM model to support a new unseen language
Home Page: https://arxiv.org/abs/2212.09535
License: Apache License 2.0
Extend GPT-2 to Korean and German by
No longer makes sense to add task-adapters since we are comparing against different parameter-efficient FT methods.
Prefix-Tuning is already supported in adapter-transformer. Just need to make it work for BLOOM.
LoRA already exists in adapter-transformer. Just need to make it support the BLOOM model.
I am getting different results by running training/eval together and separately.
Rerunning evaluation after training (by removing --do_train
) gives me a better result than running training+eval together.
The idea of this issue to modify the megatron-deepspeed repository code that we use for training all models. In order to track the progress of validation loss on several validaiton sets separately. This would allow us to track the progress of training independtly on separate languages.
Currently, the validation loss is calculated on a single validation set that includes the same language combination as the training data. (see here 13B param model training on tensorboard)
The following info is for Bloom-1.3B and embedding-and-MADX-adapters (with replace strategy) with the default bottleneck reduction size of 16.
Total frozen parameters: 1208602624
Total trainable parameters: 24979456
Total emb parameters: 20488192
Total MAD-X adapter parameters: 4,491,264
Follow discussion here
To allow comparable preplexity values across languages. We need multilingual or monolingual models that are loaded with multilingual vocabulary.
We've provided several scripts for pretraining both BERT and GPT in examples directory,
as well as scripts for both zero-shot and fine-tuned downstream tasks including MNLI, RACE, WikiText103, and LAMBADA evaluation. There is also a script for GPT interactive text generation.
Paper: https://arxiv.org/pdf/2111.09839.pdf
This will be relatively easy to add to our code.
@vnikouliNLE It seems like we are not fine-tuning the positional embedding layers transformer.wpe.weight
. I believe we should also train it too?
Create the Supervised and 0-shot Evaluation for XNLI, PAWS-X, and XQuAD
Paper:
https://aclanthology.org/2022.acl-long.125/
Implementation from paper here:
https://github.com/cambridgeltl/composable-sft
I’ll work on this starting tomorrow.
Seeing something very strange when I try to train with pfeiffer+inv adapters. I'll try to replicate this after some more GPUs free up on my end, and no problem if no one can check this (probably hard to replicate and I'd bet it'll mysteriously go away at some point :P)
{'loss': 1604337.28, 'learning_rate': 0.0009997200000000002, 'epoch': 0.02}
then loss = 0.0
by the next logging period, at 1k steps.
This is using this script (below) and the current main branch. It happens the same way both with commit f55ab013599088a35c87a880ba13a6d912e27ef4
and the current bloom
branch.
# axis
LANG="de"
MAX_TRAIN_SAMPLES=100_000
BIGS_MODEL="bigscience/bloom-1b3" # "/users/zyong2/data/zyong2/huggingface/bigscience/tr5b-1B3-multilingual-alpha-checkpoints"
ADPT_REDUCTION_FACTOR=16
adapter_config="pfeiffer+inv"
ADPT_STRAT="emb-and-adpt"
EMB_STRAT="overlap-replace"
tokenizer_dir=./trained_tokenizers//tok_bloom-1b3_de_oscar_100000samples_24000vocab_replace
cache_dir=./cache #"/users/zyong2/data/zyong2/huggingface/"
output_dir="./KEEP_RESULTS/de/1b3-postlora-"$adapter_config"-"$MAX_TRAIN_SAMPLES"samples-"$EMB_STRAT"-"$ADPT_STRAT-"$ADPT_REDUCTION_FACTOR"reduction""
logging_dir="./KEEP_RESULTS/de/1b3-postlora-"$adapter_config"-"$MAX_TRAIN_SAMPLES"samples-"$EMB_STRAT"-"$ADPT_STRAT-"$ADPT_REDUCTION_FACTOR"reduction""
mkdir -p $output_dir
mkdir -p $logging_dir
cp ./run_clm_adpt_verify.sh $output_dir/run_clm_adpt.sh
# CUDA_VISIBLE_DEVICES=4 python ../../../dev/multilingual-modeling/scripts/lang_adapt/madx_run_clm.py \
CUDA_VISIBLE_DEVICES=4 python ./madx_run_clm.py \
--fp16 \
--seed 0 \
--model_name_or_path $BIGS_MODEL \
--tokenizer_name $tokenizer_dir \
--dataset_name oscar \
--cache_dir $cache_dir \
--dataset_config_name "unshuffled_deduplicated_$LANG" \
--logging_dir $logging_dir \
--report_to "tensorboard" \
--learning_rate 0.001 \
--do_train \
--do_eval \
--output_dir $output_dir \
--preprocessing_num_workers 8 \
--overwrite_output_dir \
--per_device_train_batch_size 1 \
--gradient_accumulation_steps 8 \
--per_device_eval_batch_size 1 \
--eval_accumulation_steps 8 \
--eval_steps 5000 \
--logging_steps 100 \
--evaluation_strategy "steps" \
--max_eval_samples 5000 \
--save_steps 25000 \
--save_strategy "steps" \
--max_train_samples $MAX_TRAIN_SAMPLES \
--max_steps 50000 \
--train_adapter \
--lang_adapt_strategies $ADPT_STRAT \
--embedding_strategies $EMB_STRAT \
--adapter_reduction_factor $ADPT_REDUCTION_FACTOR \
--adapter_config ${adapter_config} \
--language $LANG \
--load_best_model_at_end
# --gradient_checkpointing
Reproduce the same experiments as done by @yongzx on English checkpoint with multilingual checkpoint: https://huggingface.co/bigscience/tr5a-1B3-multilingual-mt5tok-checkpoints/tree/global_step30000
Related issues:
TODOs
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.