MLQA results reproducability for XLM-R about xtreme HOT 7 CLOSED

Thanks for the question! There is an in the squad_metrics.py script of HuggingFace Transformers that does not work well for Chinese. Specifically, as Chinese doesn't use spaces, we don't need to execute the get_final_text function in this line. If you set final_text = tok_text instead only for Chinese, then you should get the same scores.

from xtreme.

@sebastianruder Thank you very much!

from xtreme.

@hit-computer hello, can you share which lines in evaluate_mlqa.py did you change to obtain the normal Chinese results? Just encountered the same problem on zh ):

from xtreme.

@sebastianruder @hit-computer After I set final_text = tok_text in squad_metrics.py, and ran predict_qa.py and eval_qa.py again, I still got the same results for Chinese on MLQA. Is there anything else I am missing? If I look at the predictions, most of them are very long, almost directly copying-pasting the context.
{"exact_match": 4.944520147946272, "f1": 18.297195642566766}

from xtreme.

@sebastianruder @hit-computer After I set final_text = tok_text in squad_metrics.py, and ran predict_qa.py and eval_qa.py again, I still got the same results for Chinese on MLQA. Is there anything else I am missing? If I look at the predictions, most of them are very long, almost directly copying-pasting the context.
{"exact_match": 4.944520147946272, "f1": 18.297195642566766}

+1

from xtreme.

@lixin4ever It works for me now. To evaluate ZH, we have to first delete this line, and then add final_text = tok_text at line 514. After modifying squad_metrics.py, run predict_qa.py to get the updated predicted file, and then we can evaluate it again.

from xtreme.

@ruixiangcui It also works for me. Thanks.

from xtreme.