Is there a bug in Self_Attn module? about self-attention-gan HOT 4 CLOSED

We have updated the 'sagan_models.py'. please check it out!

from self-attention-gan.

Yes, for now, although computational graph goes the same, there are some parts being corrected based on your opinions. Will be uploaded very soon!

from self-attention-gan.

Also, according to the paper softmax is performed on each row. Is https://github.com/heykeetae/Self-Attention-GAN/blob/master/sagan_models.py#L43 (dim=1) choosing columns instead?

@ognkrmms I agree with you in the first part (f_x * g_x). However, it does not make sense for the h_x * attn_score part to be matmul because of the softmax probability distribution, don't you think?

@ognkrmms @heykeetae do you know some theoretical explanation either for matmul or mul at the very first part?

from self-attention-gan.

Is there something wrong with the dimension the softmax is applied to as mentioned be @affromero ? Shoud it be dim=1 instead ?

Also in this case I think the transpose to attention tensor is not needed. Looking forward to your reply.

from self-attention-gan.