yahoojapan / jglue Goto Github PK
View Code? Open in Web Editor NEWJGLUE: Japanese General Language Understanding Evaluation
License: Creative Commons Attribution Share Alike 4.0 International
JGLUE: Japanese General Language Understanding Evaluation
License: Creative Commons Attribution Share Alike 4.0 International
Thank you for the great benchmark.
Amazon Reviews Corpus seems to be inaccessible.
$ wget https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_multilingual_JP_v1_00.tsv.gz
--2023-07-31 15:22:11-- https://s3.amazonaws.com/amazon-reviews-pds/tsv/amazon_reviews_multilingual_JP_v1_00.tsv.gz
s3.amazonaws.com (s3.amazonaws.com) をDNSに問いあわせています... 52.216.98.53, 52.216.41.112, 52.216.249.70, ...
s3.amazonaws.com (s3.amazonaws.com)|52.216.98.53|:443 に接続しています... 接続しました。
HTTP による接続要求を送信しました、応答を待っています... 403 Forbidden
2023-07-31 15:22:11 エラー 403: Forbidden。
and with the command from https://registry.opendata.aws/amazon-reviews-ml/
$ aws s3 ls --no-sign-request s3://amazon-reviews-ml/
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
We may be able to move to HuggingFace: https://huggingface.co/datasets/amazon_reviews_multi (also not available)
(I can not validate that I can generate the same dataset as the original one.)
Hi, thanks for publishing JGLUE.
The dtype for the JSTS label column is a string dtype.
I think that run_glue.py
determines if a task is a regression task or not by the dtype of the label column, so if it is a string dtype, it is treated as a classification task.
https://github.com/huggingface/transformers/blob/v4.9.2/examples/pytorch/text-classification/run_glue.py
In fact, fine-tuning BERT in JSTS resulted in a 26-value classification model.
(I have patched run_glue.py.)
On the evaluation of JSQuAD, the prediction JSON file is output with unicodes escaped. This makes it difficult to check model outputs.
It would be better to add ensure_ascii=False
in the following line
This change should be applied after merging #5, maybe?
Thank you for releasing JGLUE, but I could not evaluate my deberta-base-japanese-aozora. There seem two problems exist:
DeBERTaV2ForMultipleChoice
requires transformers
v4.19.0 and after, but JGLUE requires v4.9.2DeBERTaV2TokenizerFast
) are not supported on JSQuAD with --use_fast_tokenizer
I tried to force v4.19.2 for the problems, but I could not resolve the latter. Please see detail in my diary (written in Japanese). Do you have any idea?
I tried to reproduce the baseline score with run_squad.py
parameters you provided and patched transformers v4.19.2.
but the result score in eval_results.json
is quite low compared to the baseline.
"exact": 42.30076542098154,
"f1": 42.390814948221525,
based on fune-tuning/README.md, I think you confirmed that transformers v4.19.2 worked.
How was the score then?
I'm attaching the requirements.txt and eval_results.json when I tested with transformers v4.19.2.
Recently I found that JSNLI is (unofficially?) converted to huggingface/datasets formats (https://huggingface.co/datasets/shunk031/jsnli ), which seems really helpful.
I wonder if we could make JGLUE compatible with huggingface/datasets in the same manner...
Hi, could you provide an approximate release date of test datasets and the leaderboard? Thank you!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.