Create an integration test case based on the XING dataset about fairsearch-fair-for-elasticsearch HOT 8 OPEN

code has been uploaded to master to load the xing dataset check https://github.com/fair-search/fairsearch-elasticsearch-plugin/blob/master/demo/load-xing-dataset.rb for details.

from fairsearch-fair-for-elasticsearch.

Also a dummy query used to check manually the operations using this dataset has also been pushed to master, check https://github.com/fair-search/fairsearch-elasticsearch-plugin/blob/master/demo/xing.query

from fairsearch-fair-for-elasticsearch.

@chatox @tsuehr it would be nice to have a list of query (term, precision, significance and k) with the amount of results expected to be returned for the algorithm?.I am not sure I can find this kind of information from the paper.

from fairsearch-fair-for-elasticsearch.

This need to be constructed synthetically. Example:

query = hello

doc1 = hello hello hello hello
doc2 = hello hello hello bye
doc3 = hello hello bye bye
doc4 = hello bye bye bye
doc5 = bye bye bye bye

Now, by assigning different genres to genre1 ... genre5, one can generate expected result lists in different orderings. This depends on table p.

I suggest not to tie this to the German credit score dataset, but instead do it generically with a synthetic examples such as the one I've shown.

from fairsearch-fair-for-elasticsearch.

that works for me, we can also do that. Would you be so nice to prepare a dummy test set, including expected number of answers (per protected category) that we can translate into an integration test in the plugin? just to make sure we do the right verifications.

from fairsearch-fair-for-elasticsearch.

Sure that would be based in some mtable prepared by @tsuehr

from fairsearch-fair-for-elasticsearch.

@chatox test according to what we have spoken and what you teach us here in this issue has been created and pushed at

https://github.com/fair-search/fairsearch-elasticsearch-plugin/blob/master/src/test/java/com/purbon/search/fair/FairTopKTests.java

https://github.com/fair-search/fairsearch-elasticsearch-plugin/blob/master/src/test/resources/rest-api-spec/test/rescore/20_rescoring.yml

will also add next days more edge cases with few protected elements vs lots of protected, etc...

from fairsearch-fair-for-elasticsearch.

more tests has been added 806b804

from fairsearch-fair-for-elasticsearch.