Not correctly understanding the Multi Head Attention part of the ViT implementation... about vit-pytorch HOT 3 CLOSED

is this really MSA? are we just linearly projecting to a greater dimension specified by hyperparameters and then projecting back to the embedding dimension?

For some reason you missed those lines from Attention.forward:

qkv = self.to_qkv(x).chunk(3, dim = -1)
q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = self.heads), qkv)

dots = torch.matmul(q, k.transpose(-1, -2)) * self.scale

attn = self.attend(dots)
attn = self.dropout(attn)

out = torch.matmul(attn, v)

So it is not just linear projection to a greater dimension and back. Basically it does exactly what attention should do according to formula:

First, it computes similarity between keys and queries, then takes softmax (non linearity) to get attention weights and then applies them to values.

pytorch's MultiHeadAttention is definitely more verbose, it supports various mask, which are needed if you need causality (decoder) or you have sequences of different length, or it could use some cuda optimized hardware implementation, but in the simplest form that's basically the attention.

from vit-pytorch.

@stas-sl thank you Stas

@JavierUrenaPhDProjects i would not recommend pytorch's MHA. you will quickly find it inflexible if you want to do anything out of the ordinary

from vit-pytorch.

closing because not an issue

from vit-pytorch.