FLOPs calculation about splade HOT 6 CLOSED

hi @namespace-Pt
Sorry for the late answer!

I am not sure I completely got your point. For the FLOPS estimation, we rely on the derivation from Minimizing FLOPs to Learn Efficient Sparse Representations. The probabilities are directly estimated from the length of the posting lists (for both documents and queries, where for the later we simply "index" them).

let us know if you need more details,
Thibault

from splade.

Thank you @thibault-formal! I'll read the paper.

BTW, do you have a checkpoint of SPLADEv2-distill? I tried to reproduce your result but failed. I found that using the distillation we donot need a ground-truth passage to each query, so did you use the queries that are absent from qrels.train.tsv to train the student model?

from splade.

I would appreciate it if you could also provide the SPLADEv2-max (MRR@10=0.34) checkpoint. I want to see how the model learns to distribute the tokens.

from splade.

Hi @namespace-Pt ,

• We have some models available here: https://europe.naverlabs.com/research/machine-learning-and-optimization/splade-models/. New models will come soon, the SPLADEv2-max is not there yet but we can add it even if we have better models than this one trained without distillation.
• For your original question on the FLOPS, I understood that you were asking whether we should measure the FLOPS (Q*D) versus FLOPS(Q) + FLOPS(D). We have tried that and it did not change much the results. Sometimes, it is better to adjust differently the query and document representation.

from splade.

@sclincha @namespace-Pt the weights for SPLADEv2-max can actually be found in the weights folder in this repo.

from splade.

@sclincha @thibault-formal Thank you! I'll check it out.

from splade.