This repository contains codes for the ACL 2024 Oral paper: Prompt Optimization via Adversarial In-Context Learning
openai 0.27.4
rouge-score 0.1.2
numpy 1.22.3
transformers 4.28.1
We provide example formats of the input dataset in the folder tasks
.
-
When testing on some tasks, you can run codes in
eval/adversarial/[mathod].py
bypython gan_chat_mmlu.py (openai key) (start index of test tasks) (end index of test tasks),
โ where the index of task is in splits
. To align with the format, for flan-t5 model, you also need to input random string to alternate openai key.
-
The results will be recored in the correspoinding
.json
file. -
To evaluate the results, please run evaluation code in
eval/automatic/evaluation.py
andeval/leaderboard/create_reference_file.py
.'create_reference_file.py' is used to create labeled test dataset for tasks that you want to test by
python create_reference_file.py (start index of test tasks) (end index of test tasks)
'evaluation.py' is used to compare your anwer with labeled test dataset by
python evaluation.py --prediction_file=[address of prediction file] --reference_file=[address of reference file got by create_reference_file.py]