Code for Visual Transformer or Vision Transformer? about visualtransformers HOT 3 CLOSED

Hi there,
Thanks for your interest. There are albeit minor differences but I can assure you none of these will result in significant deviation in inference. Again thanks for your pointer and I will look to tighten the gap in future repos.

from visualtransformers.

Thanks for your reply! I still have some questions regarding the original paper of Visual Transformer, hopefully you could help me out.
1.In the paper, they set the token channel size to 1024. However, according to Fig.2 and the residual adding in equation 3 and 4, I assume the channel size is always the same with input feature map channel size (for ResNet18&34, the channel size is 256).
2. They adopt transformer encoder, however it seems to me that no multi-head attention is used from equation 3.
I appreciate it very much if you could help me out.

from visualtransformers.

Never mind, the first version paper have help me out.

from visualtransformers.