Extend GPT-2 to Korean and German by

1. Train MAD-X Language Adapters with pre-finetuned embedding layers.
2. Train MAD-X Language Adapters along with GPT-2 embedding layers.

No longer makes sense to add task-adapters since we are comparing against different parameter-efficient FT methods.

Prefix-Tuning is already supported in adapter-transformer. Just need to make it work for BLOOM.

LoRA already exists in adapter-transformer. Just need to make it support the BLOOM model.

I am getting different results by running training/eval together and separately.
Rerunning evaluation after training (by removing --do_train) gives me a better result than running training+eval together.

The idea of this issue to modify the megatron-deepspeed repository code that we use for training all models. In order to track the progress of validation loss on several validaiton sets separately. This would allow us to track the progress of training independtly on separate languages.

Currently, the validation loss is calculated on a single validation set that includes the same language combination as the training data. (see here 13B param model training on tensorboard)

Useful pointers

• How datasets are loaded in model pre-training here
• Dataset loader for GPT here
• Validation step execution here

Progress

• Forked deepspeed where all development happens (ask @hadyelsahar for invitation) here
• Pull request: bigscience-workshop/Megatron-DeepSpeed#97

The following info is for Bloom-1.3B and embedding-and-MADX-adapters (with replace strategy) with the default bottleneck reduction size of 16.

Total frozen parameters: 1208602624
Total trainable parameters: 24979456
Total emb parameters: 20488192
Total MAD-X adapter parameters: 4,491,264

Paper: https://arxiv.org/abs/2012.07463
Code: https://github.com/dguo98/DiffPruning

Experiments design

Follow discussion here

pointers to code and datasets

Pretrained Models

To allow comparable preplexity values across languages. We need multilingual or monolingual models that are loaded with multilingual vocabulary.

• ...

Datasets

• Oscar dataset tokenized using GPT tokenizer
• ....

Code

• Megatron-Deepspeed finetuning see examples here

We've provided several scripts for pretraining both BERT and GPT in examples directory, 
as well as scripts for both zero-shot and fine-tuned downstream tasks including MNLI, RACE, WikiText103, and LAMBADA evaluation. There is also a script for GPT interactive text generation. 

• ...

Progress / Results

Paper: https://arxiv.org/pdf/2111.09839.pdf

Code: https://github.com/varunnair18/FISH/blob/main/transformers/examples/text-classification/run_glue_sparse_update.py

This will be relatively easy to add to our code.

@vnikouliNLE It seems like we are not fine-tuning the positional embedding layers transformer.wpe.weight. I believe we should also train it too?

Create the Supervised and 0-shot Evaluation for XNLI, PAWS-X, and XQuAD

Paper:
https://aclanthology.org/2022.acl-long.125/

Implementation from paper here:
https://github.com/cambridgeltl/composable-sft

I’ll work on this starting tomorrow.

Seeing something very strange when I try to train with pfeiffer+inv adapters. I'll try to replicate this after some more GPUs free up on my end, and no problem if no one can check this (probably hard to replicate and I'd bet it'll mysteriously go away at some point :P)

{'loss': 1604337.28, 'learning_rate': 0.0009997200000000002, 'epoch': 0.02} then loss = 0.0 by the next logging period, at 1k steps.

This is using this script (below) and the current main branch. It happens the same way both with commit f55ab013599088a35c87a880ba13a6d912e27ef4 and the current bloom branch.

# axis
LANG="de"
MAX_TRAIN_SAMPLES=100_000
BIGS_MODEL="bigscience/bloom-1b3" # "/users/zyong2/data/zyong2/huggingface/bigscience/tr5b-1B3-multilingual-alpha-checkpoints"
ADPT_REDUCTION_FACTOR=16
adapter_config="pfeiffer+inv"

ADPT_STRAT="emb-and-adpt"
EMB_STRAT="overlap-replace"

tokenizer_dir=./trained_tokenizers//tok_bloom-1b3_de_oscar_100000samples_24000vocab_replace
cache_dir=./cache #"/users/zyong2/data/zyong2/huggingface/"
output_dir="./KEEP_RESULTS/de/1b3-postlora-"$adapter_config"-"$MAX_TRAIN_SAMPLES"samples-"$EMB_STRAT"-"$ADPT_STRAT-"$ADPT_REDUCTION_FACTOR"reduction""
logging_dir="./KEEP_RESULTS/de/1b3-postlora-"$adapter_config"-"$MAX_TRAIN_SAMPLES"samples-"$EMB_STRAT"-"$ADPT_STRAT-"$ADPT_REDUCTION_FACTOR"reduction""
mkdir -p $output_dir
mkdir -p $logging_dir

cp ./run_clm_adpt_verify.sh $output_dir/run_clm_adpt.sh

# CUDA_VISIBLE_DEVICES=4 python ../../../dev/multilingual-modeling/scripts/lang_adapt/madx_run_clm.py \
CUDA_VISIBLE_DEVICES=4 python ./madx_run_clm.py \
    --fp16 \
    --seed 0 \
    --model_name_or_path $BIGS_MODEL \
    --tokenizer_name $tokenizer_dir \
    --dataset_name oscar \
    --cache_dir $cache_dir \
    --dataset_config_name "unshuffled_deduplicated_$LANG" \
    --logging_dir $logging_dir \
    --report_to "tensorboard" \
    --learning_rate 0.001 \
    --do_train \
    --do_eval \
    --output_dir $output_dir \
    --preprocessing_num_workers 8 \
    --overwrite_output_dir \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 8 \
    --per_device_eval_batch_size 1 \
    --eval_accumulation_steps 8 \
    --eval_steps 5000 \
    --logging_steps 100 \
    --evaluation_strategy "steps" \
    --max_eval_samples 5000 \
    --save_steps 25000 \
    --save_strategy "steps" \
    --max_train_samples $MAX_TRAIN_SAMPLES \
    --max_steps 50000 \
    --train_adapter \
    --lang_adapt_strategies $ADPT_STRAT \
    --embedding_strategies $EMB_STRAT \
    --adapter_reduction_factor $ADPT_REDUCTION_FACTOR \
    --adapter_config ${adapter_config} \
    --language $LANG \
    --load_best_model_at_end
    # --gradient_checkpointing

Reproduce the same experiments as done by @yongzx on English checkpoint with multilingual checkpoint: https://huggingface.co/bigscience/tr5a-1B3-multilingual-mt5tok-checkpoints/tree/global_step30000
Related issues:

• #3
• #2

TODOs

• zero-shot evaluation of multilingual checkpoint on German and Korean
• finetune embedding layer on the German/Korean Oscar dataset
• train adapter layer on German/Korean dataset