How to evaluate the pretrained graph retriever model? about learning_to_retrieve_reasoning_paths HOT 5 CLOSED

I didn't find the file hotpot_train_order_sensitive.json for training the Hotpot distractor.

from learning_to_retrieve_reasoning_paths.

I want to evaluate the graph retriever on HotpotQA. Should I just input the 'models/hotpot_models/graph_retriever' as the output_dir

I think so, but if you would like to get the results of the selected paragraphs on HotptoQA full wiki setting, running eval_main.py with the command listed in README.md and saving the intermediate results of our graph retriever by specifying --selector_results_save_path in eval_odqa.py#L178 might be easier.

can I use the pretrained model to test the HotpotQA distractor?

You can use the model trained on the full wiki setting, but we have a separate model trained on distractor data, which might not be included in the current google drive folders. We'll upload the model.

I didn't find the file hotpot_train_order_sensitive.json for training the Hotpot distractor.

I thought it's in hotpotqa_new_selector_train_data_db_2017_10_12_fix.zip. Sorry for the inconvenience if it's not. I'm downloading to see if the hotpot_train_order_sensitive.json file is currently included or not.

[updated] Sorry we've found the file is not included. We'll update the directory shortly. Thanks for the heads up!

from learning_to_retrieve_reasoning_paths.

I am interested in the Paragraph Recall and Paragraph EM performance of the graph retrieval model both trained and evaluated on distractor. Only consider the top-1 path from the beam search. Could you tell me the two numbers? I did not find them in paper.

from learning_to_retrieve_reasoning_paths.

Yes, we don't put those numbers (we have ablation study results with top one prediction, but they are evaluated on QA data). We have some numbers from our slightly older models, and in distractor setting, the gap was actually not as large as in full wiki setting, possibly because the paragraph selections in distractor is fairly easy to the graph retriever.

from learning_to_retrieve_reasoning_paths.

I close this issue now, but please let me know if you have any followup questions!

from learning_to_retrieve_reasoning_paths.