Unbiased_lambdamart ndcg is low than original lambdamart about unbiased_lambdamart HOT 3 CLOSED

Sorry for the incomplete README and some bugs in the config file before...

(1) The original training directory is mistaken. the train_data should be "train_data = ../test_data/letor.user_browsing_model_0_train". (I've already fixed it).
(2) The data in generate_data are synthetic click data, which is not that meaningful for evaluation. What we really interested in is the result for real data. Therefore, after we get the model output result, we need to:

cd ../scripts
python eval.py ../configs/LightGBM_predict_result.txt  #or any other model output.

to do evaluation using trec (I've also added in the README now).

Also by the way, the experiment of the big table in my paper is the one not using regularization (L-0). The default regularization term in this Github is 0.5 (L-0.5), which will lead to a little bit lower result, but still much better than the original lightgbm without debiasing (which can be shown in figure 3.b).

Here is my evaluation output for your reference, the NDCG@10 with L-0.5 regularizer is 0.7535, while the original lightgbm without debiasing in my previous experiment is 0.716:

And if you want to re-implement the result of our table, you can change the regularization term back to 0, by modifying

double _eta = 1.0 / (1 + 0.5); 

in the src/objective/rank_objective.hpp (Line 418) to

double _eta = 1.0 / (1 + 0); 

Sorry for the mistakes... And hope it solves your problems.

from unbiased_lambdamart.

Thanks a lot.
I have use eval.py to evaluate test file and get original lightgbm is lower than Unbiased_LambdaMart on NDCG@10.

from unbiased_lambdamart.

Yes, I have checked trec's document and seen ndcg cal is diff.
Thks!

from unbiased_lambdamart.