about the gamma parameter about self-attention-gan HOT 18 OPEN

Based on my understanding. Self attention operation should like this : why you choose this method to calculate attention scores.

class SelfAttention(nn.Module):

def __init__(self, in_channel):
    super().__init__()
    self.query = nn.Conv1d(in_channel, in_channel // 8, 1)
    self.key = nn.Conv1d(in_channel, in_channel // 8, 1)
    self.value = nn.Conv1d(in_channel, in_channel, 1)
    self.gamma = nn.Parameter(torch.tensor(0.0))

def forward(self, input):
    shape = input.shape
    flatten = input.view(shape[0], shape[1], -1)
    query = self.query(flatten).permute(0, 2, 1)
    key = self.key(flatten)
    value = self.value(flatten)
    query_key = torch.bmm(query, key)
    attn = F.softmax(query_key, 1)
    attn = torch.bmm(value, attn)
    attn = attn.view(*shape)
    out = self.gamma * attn + input

    return out

from self-attention-gan.

We have updated the whole self attention module. please check it out! memory problem solved, and we are convinced it should agree with the paper too.

from self-attention-gan.

In this implementation, can you get better performance than the previous method you use? and can you tell me the final gamma value you trained ?

from self-attention-gan.

The performance, in honesty, is not distinguishable by human eyes. We should try the IS or FID to quantify the performance. About the gamma, the intent of the original authors goes unclear, which keeps increasing (or decreasing) under this implementation. It does not seem to converge for now, but one can try longer training to find it out!

from self-attention-gan.

Thank you very much for your comment! I think you are right about gamma. Let me make sure to correct it and update it.
About the attention map, it has the dimension of batchsize x number_of_feature (o in paper, which I interpreted as the total pixel number). It is the same as batchsize x H x W. In the code, H =W (=f in code, perhaps is the source of confusion). Since each pixel owns an attention map, requiring the total required dimension of batchsize x f^2 x f x f.
Sorry for the confusing notation. Please point out if there are still other mistakes.

from self-attention-gan.

Great suggestion! I'll sleep on that. However, that way is similar to what i tried at first, where I realized it makes more sense to have each pixel look at the different location of the previous layer by having a different attention map, since there are n number of resulting features o_j.

from self-attention-gan.

And : f_ready = f_x.contiguous().view(b_size, -1, f_size ** 2, f_size, f_size).permute(0, 1, 2, 4, 3) . Why you choose to transpose f_ready and multiply f_ready with g_ready. (why you choose transpose here)

from self-attention-gan.

That part is to reflect f(x)^T * g(x) in the paper :)

from self-attention-gan.

This operation aim to get a scalar value vector^T *vector = scalar value ; but the transpose operation in your code doesn't have this affect.
This is just my understanding.

from self-attention-gan.

That's a very good point. The calculation involves the depth of a feature map, so multiplication does not end up with a scaler (per pixel), but looking at the line, attn_dist = torch.mul(f_ready, g_ready).sum(dim=1).contiguous().view(-1, f_size ** 2), there is .sum(dim=1) following the multiplication, that sums up depth-wise, making it a scalar, per pixel.

from self-attention-gan.

If every pixel has its own attention map, the memory will be consumed quickly as the image size goes up. I agree with @Entonytang's interpretation.

from self-attention-gan.

@leehomyc I'm still not sure if having only one attention score map justifies it. Looking at the paper, Figs. 1 and 5 show attention results, where a 'particular' area takes hint from different region. In @Entonytang 's implementation, that sort of visualization is not possible.

from self-attention-gan.

I think the attention score by @Entonytang is agree with Han's paper. But, based on my understanding, attn = torch.bmm(value, attn) should like this
value = value.permute(0, 2, 1)
attn = torch.bmm( attn, value)
attn = attn.permute(0,2,1)
What do you think? @Entonytang, @heykeetae

from self-attention-gan.

why permute @hythbr

from self-attention-gan.

According to the Eqn.(2) in paper, I think the matrix-matrix product after permute may represent the meaning of the Eqn.. However, I am not sure if it is right. Please point out if there are some errors. @leehomyc

from self-attention-gan.

@Entonytang @heykeetae Hi, I read the code and doubt that how gamma change during training. It is defined as self.gamma = nn.Parameter(torch.zeros(1)) in line39 of sagan_model.py

from self-attention-gan.

Well, I have figured out that gamma is treated as a learnable parameter.

from self-attention-gan.

Related

from self-attention-gan.