The Evaluation of the index and model doesn't match of the reported paper about jpq HOT 4 CLOSED

Hi, I'm very happy to help!
I wonder whether you could reproduce the results using the code from this repo and whether the results on msmarco dev set is expected.
I will rerun the experiments today to check the code :). I'll share the ranking files once the check is over.

Best,
Jingtao

from jpq.

Hi Buruo,

I can get the reported results using the code in this repo. I don't know why you couldn't. Maybe you can provide me more details so we can work it out.

I open source the ranking results per your request.

I also provide several useful scripts to help you reproduce our work.

• Download data: download_index, download query encoder, download trec_eval
• Evaluate open-sourced checkpoints: run_retrieve.sh.

Could you please follow the guidelines in the readme and see whether run_retrieve.sh will give you the expectant numbers?

Jingtao

from jpq.

Hi Jingtao,

Thanks for the improved repository and the results file. We can confirm that your improved repository can be reproduced using your code.

Buruo

from jpq.

Great!
Looking forward to see a PyTerrier plugin for JPQ!

from jpq.