gt4sd / multitask_text_and_chemistry_t5 Goto Github PK

Nice work!
I have some problems of the metrics used in the table1, especially about the task of text2text(paragraph-actions).
You have mentioned that “For the forward prediction task the metric is accuracy; for the retrosynthesis task the metric is roundtrip accuracy (Schwaller et al., 2020); for all the other tasks the BLEU score.”
So what metric do you use in the task of text2text(paragraph-actions)? BLEU-2 or BLEU-4?

Hi, thank you for sharing this interesting work.
I'm wondering if the method still requires a fine-tuning process? As I understand it, a unified model is obtained by mixing 4 different tasks in the pre-training stage, and then inference can be performed on the test set directly without fine-tuning. However, the Code Availability section in the paper mentions that "model is available for inference, training and finetuning via the GT4SD library". So I am a little confused about the process of training, fine-tuning and inference on the test set.

How to do with this problem? I put the demo finetune code into run.sh

(gt4sd) a100-5:~/multitask_text_and_chemistry_t5$ bash run.sh

Traceback (most recent call last):
File "xxx/anaconda3/envs/gt4sd/bin/gt4sd-trainer", line 8, in
sys.exit(main())
File "xxx/anaconda3/envs/gt4sd/lib/python3.8/site-packages/gt4sd/cli/trainer.py", line 185, in main
pipeline().train(**config)
TypeError: train() missing 3 required positional arguments: 'pl_trainer_args', 'model_args', and 'dataset_args'

Hello,

I'm curious about the data splits from the ChEBI-20 dataset that were used for training the huggingface model below.
https://huggingface.co/GT4SD/multitask-text-and-chemistry-t5-base-augm

Were the test and validation sets included, or was the training set only?

Thanks for clarifying,
Sangyeup

Issue originally opened here: GT4SD/gt4sd-core#229

From Jenonone:

I cannot reproduce the results of the paper through the weights on the huggingface repo:GT4SD/multitask-text-and-chemistry-t5-base-augm ,whose accuracy is supposed to 0.322,but I test it only to be 0.196.

From medicine-wave:

Same question, I cannot reproduce the paragraph2action results of the paper through the weights on the huggingface repo: https://huggingface.co/GT4SD/multitask-text-and-chemistry-t5-small-standard, whose BLEU score is supposed to 0.929, but I test it only to be 0.659. Does the model need to be further finetuned?