Attention explanation about speech HOT 2 CLOSED

Here are two references you can take a look at:

• Attention-Based Models for Speech Recognition
• End-to-End Attention-based Large Vocabulary Speech Recognition

The attention (NNAttention) is mostly the same. The only difference is it doesn't do a linear multiply before the nonlinearity. I don't think that matrix multiply is necessary there as you can add more transformations to the encoded and decoded state in the encoder and decoder respectively. However, I haven't gotten around to testing this rigorously yet. (I expect it would be minor to no difference).

As for LinearND this is just a helper layer to do a linear transformation on something with the shape [batch, time, hidden dim] to reshape it to [batch*time, hidden dim] before the matrix operation.

from speech.

I appreciate your help very much, thank you.

from speech.