Relative Positional Encoding about axial-deeplab HOT 2 CLOSED

Thanks for your comments and suggestion.

Your re-formulation is correct. But I don't see how
y = sigma( softmax( q * k + q * r^k + k * r^q + r^q * r^k ) * (v + r^v) )
is different from
y = sigma( softmax( q * k + q * r^q + k * r^k ) * (v + r^v) )

because

1. The r^k you mentioned in the two papers is called r^q in our paper.
2. The notation of r^k and r^q does not matter because they are simply two learnable vectors. You could use any name for them.
3. The additional bias term r^q * r^k is simply an input-agnostic relative positional bias. And I don't think omitting this term makes much difference.

In addition, we were not aware of the Self-Attention with Relative Position Representations when we formulated it. So we did not follow the tradition that relative positional embeddings are added to q or k. Instead, we focused directly on the unfolded formulation and added k dependent terms to it. And we simply call the vector that will be multiplied with k, r^k.

from axial-deeplab.

Thanks for your reply!

Indeed, there is no difference between them in the implementations.

Thanks again.

from axial-deeplab.