Loading multi-file pytorch_model.bin files (eg. Mistral)? about rust-bert HOT 2 OPEN

Hello @cicero-ai ,

I would look in the direction of https://github.com/huggingface/transformers/blob/d1d94d798f1ed5c0b5de9a794381aeb7dc319c12/src/transformers/modeling_utils.py#L4082 to see how Python libraries are doing it. It seems indeed the files are not combined into a single file but rather opened sequentially and loaded in the model (the example given uses safetensors, I am unsure if you can open the individual shard files you are considering and get a valid weights dictionary).

One approach could be to create a new Resource pointing to a folder and a regex/file pattern for the archive files. You could then extend the load_weights method to handle this resource, looping through the matching shards and loading them into the model

from rust-bert.

Great, that should be all the direction I need and thank you very much for your time. One way or another it's imperative I get this working, so you'll have a PR shortly. Whether or not you want to merge it is up to you.

Actually, while I'm here one more thing -- word embeddings. Sentence embeddings work great, but I can't find a single word embeddings LLM that has a vocab resource for obvious reasons. I'm sure I can figure this one out myself and will also include PR for it, but since I'm here, if you have any directional advice on implementing word embeddings, I'm all ears.

Cheers,
Matt

from rust-bert.