Qrels and Run query ids do not match about ranx HOT 3 CLOSED

Hi @EmotionFever and thank @cadurosar,

As @cadurosar told you, this is not a bug, it is a sanity check to let you know your qrels and run query ids to not match, thus you have results for queries without relevance judgements or queries without results, or both.

Maybe, you should just remove the queries you removed from qrels from the run too.

Please let me know if this solve your issue and, if you like it, consider giving ranx a star.

Thank you,

Elias

from ranx.

Does your run contains all the elements on the qrel (and vice-versa)? I had this problem many times working with a dataset that had many queries that did not have relevance labels. It can also happen in the opposite sense (although rarer), if you have relevance data for a query that is not evaluated by your system it will also crash (reading your additional context it could be this case if you have the same qrel file for train and test)

from ranx.

Closing for inactivity. Feel free to re-open.

from ranx.