loss increasing with larger input range about fairseq HOT 6 CLOSED

Those values worked well based on cross validation experiments.

from fairseq.

为什么要乘这个数？

from fairseq.

That multiple may enlarge the value of both words and positional embedding Matrix.

from fairseq.

这大哥。。。我还以为author不睡觉回复这么快。。。
我知道会enlarge值域，但是后面应该怎么调呢？我想知道该加layer_norm还是咋地。

About why multiply sqrt(512).

Explain: I'm testing position encoding from transformer on fairseq-py. Here is my result:

Experiment 1: x = embed(tokens) * sqrt(512) + position encoding ---- loss exploding
Experiment 2: x = embed(tokens) + position encoding/sqrt(512) ---- loss decreases normal
Experiment 3: x = embed(tokens) * sqrt(512)----loss exploding
Experiment 4: x = embed(tokens) + position encoding ---- loss exploding

It's obviously the range problem.

from fairseq.

Yes, you are changing the range of the embeddings which is the problem. If you want a fixed encoding for the positions, then try to set it so that the range of the position encoding is similar to how nn.Embedding is initialized, i.e., similar to normal(0, 0.1). Also, you may want to use a custom layer instead of nn.Embedding which simply outputs those fixed position features.

from fairseq.

Thank you. Is there any explanation on why input should be in a small range (normal(0, 0.1)) instead of normal(0, 1)?

from fairseq.