Mismatch Error When Reloading Pre-trained Weights for Inference Testing about multimodal-math-pretraining HOT 4 CLOSED

@whoisfrankyang Thank you for reaching out and for your interest in the SNIP project!

After replicating the steps from the repository, I didn't encounter this error.
Noticed that you're probably running the wrong command of 1d for loading 10d model. The parameters like max_input_dimension affects data generation, vocabulary size and thus, input embedding layers in the model architecture. To load snip-10dmax.pth model, set max_input_dimension 10 as suggested in the last line of run_pretrain.sh. Can you run this command and let me know if you still encounter the mismatch error?

python train.py --reload_model ./weights/snip-10dmax.pth --loss_type CLIP --dump_path ./dump --max_input_dimension 10 --latent_dim 512

from multimodal-math-pretraining.

This command worked!
After running this command, I noticed that it is still doing pre-training work based on the given checkpoint. In the paper, it says the model runs for about 220 epochs with 1000 iterations/epoch and batch size 256. Is the checkpoint snip-10dmax.pth the result of the model for running about 220 epochs? Since the training it still going on after I reload the checkpoint, would the training ever converge? And if so, do you have an estimation on how many epochs that will take. These information can give me a rough estimate on the training time. Thanks!

from multimodal-math-pretraining.

@whoisfrankyang Happy to hear that!
Yes, the released checkpoint is after about 200 epochs. If you continue training from there, you'll likely see minimal loss reduction, indicating near convergence.

Training time varies with your setup and task specifics. For reference on training times per epoch with our setup, you can check App B.

from multimodal-math-pretraining.

@parshinsh So from my understanding, there is not really a "convergence point" in this model where the training will just stop after that. The model can technically run to the max_epoch if we let it run.

from multimodal-math-pretraining.