Discussion around BERTserini paper about cdqa HOT 4 CLOSED

What I understand from the quote below is that they only use the logits (score) for comparison between answers spans, instead of using the probabilities after applying the softmax function.

to allow comparison and aggregation of results from different segments, we remove the final softmaxlayer over different answer spans.

@fmikaelian what do you think?

from cdqa.

Here are my takeways:

• They use Anserini as their document retriever, based on open source Lucene. It uses the BM25 ranking function.
• There are 2 types of retrievers: single-stage vs. multi-stage.
• Article retrieval underperforms paragraph retrieval by a large margin.
• To score predictions they use a weighted linear interpolation between the BERT and Anserini scores: S = (1 - μ) * S_anserini + μ * S_bert with μ=0.5
• In production mode they retrieve k=10 paragraphs.
• Answering a question with this setup takes 2,35 seconds in average with a Tesla P40 GPU.

They modified BERT to compare predictions in a meaningful way (see #36):

to allow comparison and aggregation of results from different segments, we remove the final softmaxlayer over different answer spans.

But this is to be clarified.

from cdqa.

Yes

It would be useful to cross check with the paper of danqi chen where she mentions something about it with their DrQA app
https://cs.stanford.edu/~danqi/papers/thesis.pdf

Also we should follow this thread: huggingface/transformers#360

from cdqa.

In section 5.2.3 of Danqi Chen's thesis:

We apply our trained DOCUMENT READER for each single paragraph that appears inthe top 5 Wikipedia articles and it predicts an answer span with a confidence score. To make scores compatible across paragraphs in one or several retrieved documents, we use the unnormalized exponential and take argmax over all considered paragraph spans for our final prediction. This is just a very simple heuristic and there are better ways to aggregate evidence over different paragraphs

This part seems to be implemented here:

https://github.com/facebookresearch/DrQA/blob/d27180fc527084263ca0e43091f5d35c4bbd4963/drqa/reader/layers.py#L243

And here:

https://github.com/facebookresearch/DrQA/blob/1f811ded549a69f8b5ea303fb6f6d35ad6fc84ae/drqa/pipeline/drqa.py#L113

Our predict() function does not currently returns this confidence score. How can we get it in our setup and modify it for comparision?

from cdqa.