Any hyper parameter suggestions for other model architectures? about moco-v3 HOT 3 CLOSED

Yeah it is quite sensitive to the hyper parameters in linear probing, which we also observed. One idea we adopted in our more recent MAE work (https://arxiv.org/abs/2111.06377) is to add an additional, parameter-free BN0 (so BN without learnable weights) to normalize features "on-the-fly" before the linear classifier. Since the BN0 stats can be absorbed in the weights, it does not violate the linear probing rule and can help reduce parameter search when changing to other architectures. For linear probing, the main parameters to search is learning rate (likely due to different feature scales).

from moco-v3.

I read the MAE recently, it's a really wonderful work!

But i'm very confused about the handling of class token when pre-train the MAE model.

The paper said: As ViT has a class token [16], to adapt to this design, in our MAE pre-training we append an auxiliary dummy token to the encoder input. This token will be treated as the class token for training the classifier in linear probing and fine-tuning.

How can this dummy token work in linear probing ? Since it's not explicitly used in pre-training. Is there any other information I missed?

from moco-v3.

I read the MAE recently, it's a really wonderful work!

But i'm very confused about the handling of class token when pre-train the MAE model.

The paper said: As ViT has a class token [16], to adapt to this design, in our MAE pre-training we append an auxiliary dummy token to the encoder input. This token will be treated as the class token for training the classifier in linear probing and fine-tuning.

How can this dummy token work in linear probing ? Since it's not explicitly used in pre-training. Is there any other information I missed?

Hello, Have you tried other architectures like EfficientNet?

from moco-v3.