will it make any sense to use zero v in the first decoder layer? about detr HOT 6 CLOSED

Your understanding is correct. The first decoder self attention does not receive any data dependent inputs, so we pass zeros as inputs. It could be removed to save some parameters and compute, but we keep for simplicity.

We should add a comment explaining this in the code.

from detr.

Your understanding is correct. The first decoder self attention does not receive any data dependent inputs, so we pass zeros as inputs. It could be removed to save some parameters and compute, but we keep for simplicity.

We should add a comment explaining this in the code.

As a matter of fact, not only in the first decoderlayer, but also all the subsquent layers, the self attention part can be skipped actually. Your choice is the 3-sublayered version, a little strange because such 3-sublayered decoder are more used in autogressive tasks like seq2seq. A 3-sublayered decoder will give a better performance?

from detr.

no, in subsequent layers self attention has meaningful data dependent inputs and needed for communication, so can't be skipped.

from detr.

OK, never mind for my wrong understanding.

from detr.

Hi,

I believe we have answered your question, and as such I'm closing the issue, but let us know if you have any further questions.

from detr.

@rardz @szagoruyko

tgt2 = self.self_attn(q, k, value=tgt2, attn_mask=tgt_mask, key_padding_mask=tgt_key_padding_mask)[0]

https://pytorch.org/docs/stable/_modules/torch/nn/modules/activation.html#MultiheadAttention

Here self_attn will add a projection layer with learnable bias on q, k, and v. Though the tgt2 is a zero tensor at the first layer, it will still hold non-zero values.

from detr.