Batch-dependent behavior in SoftmaxCEMaskedLoss? about gluon-nlp HOT 1 OPEN

@JulianSlzr , in the current nmt scripts, the token loss is averaged by the total number of the tokens in the mini-batch. You can imagine it by concatenating all the sequences in the mini-batch to a single one and taking average. You can find it in the following code.
loss = loss * (tgt_seq.shape[1] - 1) / (tgt_valid_length - 1).mean()
This ensures that all the tokens are equally weighted. In fact, we are minimizing the following objective:
\sum_{i=1}^N\sum_{t=1}^{T_i}\logP(y_t^i|y_1^i, ...y_{t-1}^i, X^i, \Theta)
But sure, what you propose can be useful in some scenarios. A PR is always welcome.

from gluon-nlp.