Hello,

Please help , which optimizer should i use while fine-tuning your model ? and with which learning rate ?

Thanks.

hello, i am trying to fine tune this model on arabic to english MT but i'm not getting any good results, it says in the paper that it works for this specific task but it didn't work for me i dont know why, the bleu score didnt surpass 10

Hi,

I want to run your scripts run_trainier_seq2seq_huggingface.py but it gives me the folowing error:

last_checkpoint None
03/26/2022 14:49:11 - WARNING - main - Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False
03/26/2022 14:49:11 - INFO - main - Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='/content/AraT5_FT_title_generation', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=<IntervalStrategy.EPOCH: 'epoch'>, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_ratio=0.0, warmup_steps=0, logging_dir='runs/Mar26_14-49-11_1f6d9e124699', logging_strategy=<IntervalStrategy.EPOCH: 'epoch'>, logging_first_step=False, logging_steps=500, save_strategy=<IntervalStrategy.STEPS: 'steps'>, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', fp16_backend='auto', fp16_full_eval=False, local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='/content/AraT5_FT_title_generation', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=True, metric_for_best_model='eval_bleu', greater_is_better=True, ignore_data_skip=False, sharded_ddp=[], deepspeed=None, label_smoothing_factor=0.0, adafactor=False, group_by_length=False, length_column_name='length', report_to=['tensorboard'], ddp_find_unused_parameters=None, dataloader_pin_memory=True, skip_memory_metrics=False, mp_parameters='', sortish_sampler=False, predict_with_generate=True)
[INFO] loading from TSV
03/26/2022 14:49:11 - WARNING - datasets.builder - Using custom data configuration default-942a41af4b2c6152
Downloading and preparing dataset csv/default to /tmp/AraT5_cache_dir/csv/default-942a41af4b2c6152/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519...
Downloading data files: 100% 2/2 [00:00<00:00, 9446.63it/s]
Extracting data files: 100% 2/2 [00:00<00:00, 985.16it/s]
Dataset csv downloaded and prepared to /tmp/AraT5_cache_dir/csv/default-942a41af4b2c6152/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519. Subsequent calls will reuse this data.
100% 2/2 [00:00<00:00, 803.74it/s]
Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 466, in get_config_dict
user_agent=user_agent,
File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 1173, in cached_path
local_files_only=local_files_only,
File "/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py", line 1383, in get_from_cache
"Cannot find the requested files in the cached path and outgoing traffic has been"
FileNotFoundError: Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "run_trainier_seq2seq_huggingface.py", line 807, in
main()
File "run_trainier_seq2seq_huggingface.py", line 365, in main
local_files_only = True
File "/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py", line 398, in from_pretrained
config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/configuration_utils.py", line 478, in get_config_dict
raise EnvironmentError(msg)
OSError: Can't load config for 'UBC-NLP/AraT5-base'. Make sure that:

• 'UBC-NLP/AraT5-base' is a correct model identifier listed on 'https://huggingface.co/models'

• or 'UBC-NLP/AraT5-base' is the correct path to a directory containing a config.json file

How to fix it please?

I need to fine tune the translation task should i prepare the data in specific format?
and how to fine tune your model for that task?

Hi @Nagoudi @elmadany ,

Thank you so much for open-sourcing your awesome models. I have a question please, I want to use AraT5 or AraT5v2 for machine translation from English to Arabic. Could you please share an example to do that? I tried to use your models with the following code but the output does not make sense. Here is the code:

from transformers import T5Tokenizer, AutoModelForSeq2SeqLM, AutoTokenizer, pipeline


model = AutoModelForSeq2SeqLM.from_pretrained("UBC-NLP/AraT5-msa-base")
tokenizer = AutoTokenizer.from_pretrained("UBC-NLP/AraT5-msa-base")
tokenizer.src_lang="English"
tokenizer.tgt_lang="Arabic"

ar_prompt="The scene displays a group of people gathered around a wooden dining table in an indoor setting."
input_ids = tokenizer(ar_prompt, return_tensors="pt").input_ids


outputs = model.generate(input_ids)
print("Tokenized input:", tokenizer.tokenize(ar_prompt))
print("Decoded output:", tokenizer.decode(outputs[0], skip_special_tokens=True))

This is the current output:

Tokenized input: ['▁The', '▁scene', '▁display', 's', '▁a', '▁group', '▁of', '▁people', '▁gathered', '▁around', '▁a', '▁wooden', ```
▁di', 'ning', '▁table', '▁in', '▁an', '▁indoor', '▁setting', '.']

Decoded output: هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب هحب

Please let me know what the issue is in the above code or share an example of how to use your model for translation from English to Arabic.

I'm not able to find the ARGEN benchmark datasets directly. I can see links to the sources for both pretraining and most other tasks but I was hoping to be able to access the complete benchmark at one location. Is it possible to make ARGEN_CST available ? @Nagoudi @elmadany @mageed .

Hi,

Thanks for sharing the code and models of your great paper.

I think that you miss calculating the rouge scores for Text summarization task in your paper.
The bug lies in these lines:

Let me explain. First, the rouge_score (which is embedded in HF datasets library) don't work on Arabic text. Here is a simple example were the reference and prediction are the exactly the same:

from rouge_score import rouge_scorer

gold = "اختر العمر المستقبلي."
pred = "اختر العمر المستقبلي."
rouge_types = ["rouge1", "rouge2", "rougeL"]
scorer = rouge_scorer.RougeScorer(rouge_types=rouge_types, use_stemmer=False)
print({key: value.fmeasure * 100 for key, value in score.items()}) #{'rouge1': 0.0, 'rouge2': 0.0, 'rougeL': 0}

This happen because the default tokenizer of google rouge wrapper will delete all non alphanumeric characters (see comment 2 for a solution).

However, rouge works well on English:

from rouge_score import rouge_scorer

gold = "police kill the gunman"
pred = "police kill the gunman"
rouge_types = ["rouge1", "rouge2", "rougeL"]
scorer = rouge_scorer.RougeScorer(rouge_types=rouge_types, use_stemmer=False)
print({key: value.fmeasure * 100 for key, value in score.items()}) #{'rouge1': 100.0, 'rouge2': 100.0, 'rougeL': 100.0}

gold = "police kill the gunman"
pred = "police killed the gunman"
rouge_types = ["rouge1", "rouge2", "rougeL"]
scorer = rouge_scorer.RougeScorer(rouge_types=rouge_types, use_stemmer=False)
print({key: value.fmeasure * 100 for key, value in score.items()}) #{'rouge1': 75.0, 'rouge2': 33.33333333333333, 'rougeL': 75.0}

When in your code you comments these 2 lines because they gives scores around 1%-2% (I will explain why later)

and you replace it with these 2 lines:

what you actually did is dividing the number of \\n \\n span in the reference and prediction.
Here is a simple example where the gold reference has 2 sentences and prediction have 4 sentences:

gold = "اختر العمر المستقبلي. كن عفويا."
pred = "ابحث عن العمر الذي تريد أن تكونه في المستقبل. تحدث عن نفسك الحالية. فكر في قيمك. فكر في الأشياء التي تجيدها."

# No linebreak  between senctences
score = scorer.score(gold, pred)
print({key: value.fmeasure * 100 for key, value in score.items()}) # {'rouge1': 0.0, 'rouge2': 0.0, 'rougeL': 0}

# Correct way to add linebreak  between sentences
gold_2 = gold.replace(". ", ".\n")
pred_2 = pred.replace(". ", ".\n")
score = scorer.score(gold_2, pred_2)
print({key: value.fmeasure * 100 for key, value in score.items()}) # {'rouge1': 0.0, 'rouge2': 0.0, 'rougeL': 0}


# Adding linebreak between sentences with your method
gold_3 = gold.replace(". ", ".\\n \\n ")
pred_3 = pred.replace(". ", ".\\n \\n ")
score = scorer.score(gold_3, pred_3)
print({key: value.fmeasure * 100 for key, value in score.items()}) # {'rouge1': 50.0, 'rouge2': 33.333333333333336, 'rougeL': 50.0}

print(gold_3) # اختر العمر المستقبلي.\n \n كن عفويا.
print(pred_3)# ابحث عن العمر الذي تريد أن تكونه في المستقبل.\n \n تحدث عن نفسك الحالية.\n \n فكر في قيمك.\n \n فكر في الأشياء التي تجيدها.

As you can see, in the example <gold_3, pred_3> the 50 rouge is because you predicted 4 sentences (actually \\n \\n) while the reference contains 2 only. The 33% is because you have 1 and 3 \\n \\n n-grams in the reference and prediction respectively (1/3).

In fact, I re-run Text summarization experiments internally using your model and models mine and found that the results are similarly comparable with your paper when using your method. On the other hand, when adding \n correctly the socres are between 1% and 2%. The rouge scores are not zero happen only when the reference and prediction contains same English words, whci happen rarely.

In fact, results in tables 7 and B2 are just the count(sent_num_ref) / count(sent_num_pred).
Don't understand me wrong, your models are good but just need to be evaluated correctly (see comment 2).

It will be great if you can fix your code and adjust the numbers in table 7 and B2 in your paper.

Thanks

I need to fine tune the paraphrasing task should i prepare the data in specific format?
and how to fine tune your model for that task?

I can't find any token like <extra_id> or < mask > in the vocab. What is the mask token in AraT5-base or how do I get the mask id with huggingface codes?

Hello contributors!
Thank you for the amazing project. Wonderful work.
I am working on a text classification task, and found AraT5 to be very useful for my case. I was going through Fine_tuning_AraT5.ipynb featured in the examples directory. I trained the model, following your "best results" instructions (22 epochs...) on my own data, and got class predictions not featured in the original classification classes. I then tried training on the sample data provided in the notebook (ARGEn_title_genration_sample_train.tsv), and got the following results:

Please note the prediction results were mostly the same, regardless of the training dataset (mine or the one provided).

Am I missing something? Could you please help with this? It would be much appreciated.

Hi,

Thank you for sharing your great work!

Can you tell me please how to finetune one of your pretrained LM for a Question-Answering (QA) task?

As input, I have question and context. As output one or multiple answers.

It's very urgent, please!

Thank you so much.

hello

Where can we find the operative config gin file of araT5 Base msa model. It is needed to continue the training of the model using a TPU.

the operative config gin files of the English T5 is available is available in the folder of every pretrained model such as
gs://t5-data/pretrained_models/small/operative_config.gin

Thank you

Hello, Thanks for the models.
I was trying to run the provided sample code in the README, but I got the following error:

[ValueError: Couldn't instantiate the backend tokenizer from one of: 
(1) a `tokenizers` library serialization file, 
(2) a slow tokenizer instance to convert or 
(3) an equivalent slow tokenizer class to instantiate and convert. 
You need to have sentencepiece installed to convert a slow tokenizer to a fast one.]()

any ideas what is the issue?

Hello,

I am working on a project that requires me to use the AraT5 model, and I am wondering what tokenizer was used to train the model and where I can find the vocabulary list.

I have searched the documentation and code for the model, but I haven't been able to find this information. If someone could provide me with this information or point me to where I can find it, I would greatly appreciate it.

Thank you!

Thanks a lot for the great code, just wandering if I want to do closed book question answering fine tuning, do I need to specify a particular prefix before fine-tuning or just examples of questions and answers will be enough?