How is "lilt-only-base" bin file is created about lilt HOT 3 CLOSED

Hi,
you can read our original paper at https://aclanthology.org/2022.acl-long.534/. As explained in it, LiLT-base+En-Roberta are pre-trained using English docs. And the provided "lilt-only-base" is exactly the pre-trained LiLT-base part. It can be used to combine different textual models to deal with docs in different languages during fine-tuning.

from lilt.

Hi, you can read our original paper at https://aclanthology.org/2022.acl-long.534/. As explained in it, LiLT-base+En-Roberta are pre-trained using English docs. And the provided "lilt-only-base" is exactly the pre-trained LiLT-base part. It can be used to combine different textual models to deal with docs in different languages during fine-tuning.

I understand the usage. But I am very curious how the file "lilt-only-base" is created. As you have mentioned what is "pre-trained LiLT-base part" how that specific base part is created.
We all know how roberta-en is created and from your provided code how "gen_weight_roberta_like.py" generates the base + roberta model.

What does the base part contains.

from lilt.

Pytorch uses a dict-like format to store weight name-value pairs in 'pytorch_model.bin' files. We just filter out the name-value pairs of the LiLT part by weight names from the pre-trained checkpoint to create "lilt-only-base".

from lilt.