Question about get_visual_prompt about hvpnet HOT 2 CLOSED

Thanks for your attention!

1. prompt_guids is 4 * [bsz, 256, 2, 2]. Shouldn't the dimension of torch.cat(prompt_guids, dim=1) is [bsz, 1024, 2, 2], but it's [bsz, 3840, 2, 2] in fact.

Actually, prompt_guids is [[bsz, 256, 2, 2], [bsz, 512, 2, 2], [bsz, 1024, 2, 2], [bsz,2048, 2, 2]], so the dimension of concatenation is [bsz, 3840, 2, 2].

2. What does this code do? Is it just calculating the k,v values of superimposing four resnet blocks onto that self-attention layer?

for i in range(4):
   key_val = key_val + torch.einsum('bg,blh->blh', prompt_gate[:, i].view(-1, 1), split_prompt_guids[I])

The goal of this code is to use the dynamic gates(prompt_gate) to aggregate hierarchical image features(split_prompt_guids) to obtain the visual prefix(key, values).

3. Why choose 64?

We use bert-base-uncased as the PLM, which has 768 hidden size and 12 attention heads, so the size of each attention head is768 / 12 = 64.

4. If I want to switch to roberta plm, where I need to modify?

You need to modify the source code modeling_roberta.py like models/modeling_bert.py. For example, set past_key_values_length=0 before embedding_output = self.embeddings(...). For other places that need to be modified, please refer to the content annotated with #add in models/modeling_bert.py.

from hvpnet.

Thank you very much for your answer,
Wish you a happy life！

from hvpnet.