Why using random sampling during inference and not pick instead the X patches with maximum attention? about attention-sampling HOT 1 OPEN

Hi,

It's been a while but better late than never.

It never occurred to me to actually pick the top-k locations during inference. It might indeed be beneficial in comparison to sampling. Something to keep in mind would be that this would no longer be an unbiased approximation so it might introduce some bias especially if the attention is very peaky.

Regarding, ATS-10 vs ATS-50 we use the same number of patches both during training and inference.

Having the model predict the patch size would be an interesting extension to this paper. For instance, one could predict two Gaussian random variables for patch width and height and then use the same math to approximate the average feature integrating over the patch size values.

Cheers,
Angelos

from attention-sampling.