weird attention weights when adding sequence of numbers. about keras-attention HOT 1 CLOSED

First if your loss converges to 0, that means your model is learning.
Regarding the attention weights, we can always have different explanations but the most plausible is:

• We unroll the recurrent net on the sequence. At time t, you have technically a summary of all the information from 0 until t (aim of a recurrent net).
• That means you don't need the outputs of the rec. net at time 0, 1, 2 but just time t is enough.
• The attention layer shows this. If you only take the output of the LSTM at time t, then you can solve the problem.
  So that's why the attention is maximum after the LSTM processes the delimiter. It's a mechanism to say, this value matters because it contains all the previous steps and we don't care about the values after that.

from keras-attention.