yahoojapan / jglue Goto Github PK

Thank you for the great benchmark.

Amazon Reviews Corpus seems to be inaccessible.

$ wget https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_multilingual_JP_v1_00.tsv.gz
--2023-07-31 15:22:11--  https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_multilingual_JP_v1_00.tsv.gz
s3.amazonaws.com (s3.amazonaws.com) をDNSに問いあわせています... 52.216.98.53, 52.216.41.112, 52.216.249.70, ...
s3.amazonaws.com (s3.amazonaws.com)|52.216.98.53|:443 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 403 Forbidden
2023-07-31 15:22:11 エラー 403: Forbidden。

and with the command from https://registry.opendata.aws/amazon-reviews-ml/

$ aws s3 ls --no-sign-request s3://amazon-reviews-ml/

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

We may be able to move to HuggingFace: https://huggingface.co/datasets/amazon_reviews_multi
(I can not validate that I can generate the same dataset as the original one.) (also not available)

Hi, thanks for publishing JGLUE.

The dtype for the JSTS label column is a string dtype.

I think that run_glue.py determines if a task is a regression task or not by the dtype of the label column, so if it is a string dtype, it is treated as a classification task.
https://github.com/huggingface/transformers/blob/v4.9.2/examples/pytorch/text-classification/run_glue.py

In fact, fine-tuning BERT in JSTS resulted in a 26-value classification model.
(I have patched run_glue.py.)

On the evaluation of JSQuAD, the prediction JSON file is output with unicodes escaped. This makes it difficult to check model outputs.

It would be better to add ensure_ascii=False in the following line

This change should be applied after merging #5, maybe?

Thank you for releasing JGLUE, but I could not evaluate my deberta-base-japanese-aozora. There seem two problems exist:

• DeBERTaV2ForMultipleChoice requires transformers v4.19.0 and after, but JGLUE requires v4.9.2
• Fast tokenizers (including DeBERTaV2TokenizerFast) are not supported on JSQuAD with --use_fast_tokenizer

I tried to force v4.19.2 for the problems, but I could not resolve the latter. Please see detail in my diary (written in Japanese). Do you have any idea?

I tried to reproduce the baseline score with run_squad.py parameters you provided and patched transformers v4.19.2.
but the result score in eval_results.json is quite low compared to the baseline.

    "exact": 42.30076542098154,
    "f1": 42.390814948221525,

based on fune-tuning/README.md, I think you confirmed that transformers v4.19.2 worked.
How was the score then?

I'm attaching the requirements.txt and eval_results.json when I tested with transformers v4.19.2.

Recently I found that JSNLI is (unofficially?) converted to huggingface/datasets formats (https://huggingface.co/datasets/shunk031/jsnli ), which seems really helpful.

I wonder if we could make JGLUE compatible with huggingface/datasets in the same manner...

Hi, could you provide an approximate release date of test datasets and the leaderboard? Thank you!