Fail to Reproduce the dev score of GENRE Document Retrieval about genre HOT 7 OPEN

Training seems correct. Are you using constrained search during the evaluation?

from genre.

Also, with BLINK you do not need to use convert_kilt_to_fairseq.

from genre.

Thanks for your quick response.

1. I use constrained search during the evaluation. The trie is downloaded from kilt_titles_trie_dict.pkl, and I use evaluate_kilt_dataset.py with beam=10, max_len_a=384, max_len_b=15.

2. I get the BLINK train set in JSON Line format from blink-train-kilt.jsonl. It seems that this file is structured in the same way as other KILT datasets. So I just cat the blink-train-kilt.jsonl and other 8 KILT train jsonl files mentioned in the paper to one single file. Then I shuffle this JSON Line file with random.shuffle() using python.
   I cat all 11 dev jsonl files of KILT to one file as development set.
   Then use the script convert_kilt_to_fairseq.py & preprocess_fairseq.sh to process above files.

Am I doing these the right way?

Thanks again!

from genre.

@nicola-decao

from genre.

Yes, you are doing it correctly then. I am not sure what is going wrong. Are you sure you are training with the same batch size and number of steps as reported in the paper?

from genre.

Yes, I rerun the whole finetune process on 8 V100 GPUs torch1.6.0+cuda10.1. I directly use the training script train.sh with max-tokens per GPU to 1024, update-freq to 128, max-update to 200000, which should be the same hypermeters reported in the appendix A.3. I get the following results.

T-REx, zsRE, NQ, HoPo, TQA are still lower than expected.

from genre.

That is weird, but I do not know how to help. I do not work at Facebook/ Meta anymore, so I cannot re-run experiments or check the original code that was launched. Note: I run on more GPUs.

from genre.