Isn't the softmax of the attention matrix calculated incorrectly? about vit-pytorch HOT 2 CLOSED

@PascalHbr Hello! So I believe it is correct the way it is

The image has already been cut into patches and linearized into tokens. The dimensions of i and j are actually the similarity of each patch i to every other patch j including itself. You then force the network to pick (with softmax) the patch that should be attended to. This is done on the j dimension, so that each patch i picks one of the other patches that is most important.

Perhaps you were thinking there is some correspondence between i, j and the height and width, but i and j are actually height * width.

from vit-pytorch.

Thanks a lot for the clarification. You are right indeed :)

from vit-pytorch.