• Official code for our ACL 2019 paper.
• The initial code was adapted from Adversarial Squad.
• We adapt the Corenlp code (nectar) to support python 3.
• The arithmetic on Glove word embeddings are adapted from (https://github.com/brannondorsey/GloVe-experiments).

Run download_data.sh to pull HotpotQA data and GloVe vectors.

• We tested our code on TF1.3, TF1.8, TF1.11 and TF1.13.
• See requirements.txt.

Run:

python3 convert_sp_facts.py corenlp -d dev

to store preprocessed data in data/hotpotqa/dev_corenlp_cache_***.json. This avoids rerunning Corenlp every time we generate an adversarial data. If you want to create the adversarial training data, run:

python3 convert_sp_facts.py corenlp -d train

Warning: preprocessing both the training set and dev set requires a storage space of ~22G.

Run:

python3 convert_sp_facts.py gen-answer-set -d dev

and

python3 convert_sp_facts.py gen-title-set -d dev

This step collect all answers and Wikipedia article titles in the dev set and classify them based on their NER and POS tag.

If you want to eliminate the title-balancing bias in the adversarial documents (described in the last paragraph of Sec. 2.2), run:

python3 convert_sp_facts.py gen-all-docs -d dev

To generate the adversarial dev set described in our paper, run:

python3 convert_sp_facts.py dump-addDoc -d dev -b --rule wordnet_dyn_gen --replace_partial_answer --num_new_doc=4 --dont_replace_full_answer --find_nearest_glove --add_doc_incl_adv_title

This will create the adversarial training set in out/hotpot_dev_addDoc.json Note: --add_doc_incl_adv_title can be set only if Step 3 is done.

Generating the adversarial training set all at once could take days. Therefore, we divide the training set into 19 batches with the size of 5000, and process each batch in a separate program by running:

python3 convert_sp_facts.py dumpBatch-addDoc -d train -b --rule wordnet_dyn_gen --replace_partial_answer --num_new_doc=4 --dont_replace_full_answer --find_nearest_glove --add_doc_incl_adv_title --batch_idx=0

with batch_idx set to 0~18. After they finish, run:

python3 convert_sp_facts.py merge_files -d train

This will create the adversarial training set in out/hotpot_train_addDoc.json

@inproceedings{Jiang2019reasoningshortcut, 
	author={Yichen Jiang and Mohit Bansal}, 
	booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, 
	title={Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA}, 
	year={2019}, 
}