Inquiry about Configuration Details for "ecir23-scratch-tydi-japanese-splade" Model about splade HOT 4 CLOSED

Hi @kuro96al,

we pretrained the model from scratch using the japanese Mr.TyDi corpus (https://github.com/castorini/mr.tydi), we then trained with a contrastive loss using japanese MMARCO (https://github.com/unicamp-dl/mMARCO) and finally finetuned with the japanese Mr.TyDi train query set.

The model is based on a distilbert (6L, 768Hidden dims), but as said previously, the model is initialized randomly and then trained as described in the previous paragraph.

For more information here's a paper talking about the strategies we used to develop that model and what we were looking for: https://arxiv.org/pdf/2301.10444.pdf

from splade.

Thank you for your response. Is the pre-trained model uploaded on platforms like Hugging Face?

from splade.

We attempted to train SPLADE based on the model found at https://huggingface.co/line-corporation/line-distilbert-base-japanese/tree/main, but it seems that there were issues with the vocabulary that prevented successful training.

from splade.

Thank you for your response. Is the pre-trained model uploaded on platforms like Hugging Face?

Unfortunately it is not, not sure if we still have it...

We attempted to train SPLADE based on the model found at https://huggingface.co/line-corporation/line-distilbert-base-japanese/tree/main, but it seems that there were issues with the vocabulary that prevented successful training.

Yeah, we found similar problems with a ton of models, that's one of the reasons we went with training a model from scratch.

from splade.