How can you resolve the mismatch of patch_size in <patch_embed> module between pretrained model and finetuned model? about simmim HOT 2 CLOSED

As far as I know, for ViT experiment: you use random mask with patch size 32. But when running fine-tuning, you use a default patch size of 16. I got confused on how you tackle the mismatch in <patch_size> between these. As I dig into your utils.py, the load_pretrained function only resolves the mismatch patch size for Relative Position Embeddings. Please correct me if I miss some things?

The masking patch size does not need to be equal to image patch size. In training, masking patch size of 32 and image patch size of 16 is used in ViT. In inference, there is no masking, and an image patch size of 16 is used.

from simmim.

Oh, I see. I was confused at first because I supposed that in pretraining, the role of masking patch size and image patch size is the same (namely equals to 32). Thanks for your quick response!

from simmim.