Multilingual AVSR model decoding and training about muavic HOT 2 OPEN

Hi @roudimit,

Thank you for raising this issue and so sorry for the late reply!

I fixed this by adding ...

Does this happen to the other monolingual models as well?

Did you use the language tag in the reference for evaluation in the multilingual setting?

Yes, and later on I found out it is NOT the common practice. So, we are going to update the results in a newer version of the MuAViC paper.

Is there a way to force output text in a certain language?

I never tried this before. However, theoretically we can use bos_index to do so with the multilingual dictionary (dict.x.txt) to know the index of the bos symbol. For example <ar> is the first word in the dictionary, then its index gonna be 4 since the first four indices are: <s>, <pad>, </s>, and <unk> in that order.

Should I add the language tag to the beginning of all sentences?

That's how I did it in the paper. However, looking back I don't think it was necessary.

How do you balance samples from different languages?

In the paper, we used random sampling, which doesn't balance samples from different languages. However, you can balance the dataset sampling by following these steps:

• First, create different TSV files, one for every language.
• Then, set the dataset_train_subset to these files as comma-separated, e.g. dataset.train_subset=train_en,train_ar,train_el, ...etc.
• Then, change the load_dataset to be similar to this.

from muavic.

Hi @Anwarvic thanks for the clarifications! I'll keep this issue open for now since the multilingual model WER are impacted by the language tag in the beginning, and since you are planning to update the results.

from muavic.