nDCG metric does not correctly truncate lists about lkpy HOT 4 CLOSED

Thanks for the clarification, I now understand the utility of the current implementation and agree that 1 is not a bug.

from lkpy.

#310 fixes this for a LensKit 0.14.1 bug-fix release. Will cherry-pick that fix onto the main branch and then close the bug.

from lkpy.

2 definitely looks like a bug. Will work on a fix.

Regarding 1: when k is not specified, this is actually a difficult scenario, and it is not clear what the correct behavior is, because it is not clear how many results the system was supposed to return. LensKit's current behavior assumes that the target list length is effectively unbounded (or at least as long as the truth frame), and thus a truncated list represents an inability to recommend further items. Another option is to assume that the list is as long as was requested, and truncate the truth; this is a quite strong assumption, and it means a recommender can obtain a higher NDCG score simply by not recommending more items.

Disambiguating this case is why k was introduced, and evaluations should generally always specify k (should make that clearer in the documentation, perhaps). From the data frame alone, the metric does not have enough information to determine a target length. Do you have any references to what assumptions are made by other implementations or authors when result lists are too short? Happy to discuss this further and be persuaded that this is a bug, but right now it looks like differing expected assumptions for an edge case.

from lkpy.

@spawliko thanks! I should be able to have a fix for 2 in testing later today.

from lkpy.