[BUG] Precision calculation incorrect? about ranx HOT 6 CLOSED

Hi @kaleko,

Thank you very much for the bug report and for providing a working example!
numba was not raising a ZeroDivisionError, so I did not spot this issue before.
I fixed it in v.0.3.4. Now it works as intended.

Please, consider giving ranx a star if you like it!

from ranx.

It seems that if there is an empty query result in the run_dict, every query after it will always have a precision of 0.

from ranx.

@AmenRa I now see that in the above example, the outputs are run 1 --> precision of 1.0, run 2 --> precision of 0.75, run 3 --> precision 0.75.

It's good to see runs 2 and 3 have the same precision, the result of fixing your ZeroDivisionError issue.

However I question whether the actual precision calculation is correct.
According to this comment

In all three runs above, every document which was retrieved was relevant. Shouldn't the precision be 1.0 for all runs?

from ranx.

Usually, a system does not return documents whose relevance score is zero.
That's why you could end up with empty result lists, as in your example.
However, this is probably "a convention" because 1) you cannot meaningfully order the documents if they all have the same relevance score (so the system's output would be kind of random), and 2) if the system returns the entire collection every time it is queried, it will have severe efficiency issues.

Moreover, if you cast Information Retrieval to a binary classification problem, the returned documents would be the data points judged as positives by the model and the non-returned ones as negatives.
If you have a query for which no document was returned, the model judged all the documents as negatives (non-relevant to the query).

I think returning no documents for one or more queries is a corner case.
If we take this corner case to the extreme, a system that never returns documents should have Precision=1.0 on average following the last line of your comment, which does not sound right to me.

Makes sense / do we agree?

from ranx.

I guess I agree. It sounds like a convention.

For example, if I google "awefoihawoefihawoefihw" and zero results come back, did my query have 100% precision or 0% precision? I would argue 100%, but I can see both sides.

Thanks for the clarification.

from ranx.

If you find a theoretically sound explanation, please post it here.

from ranx.