konstantinoskokos / lassy-tlg-supertagging Goto Github PK

When reporting loss, in train_epoch, the code says
batch_p = self.forward(batch_x, batch_e, encoder_mask, decoder_mask)

However in eval_epoch "infer"
batch_p = self.transformer.infer(batch_x, encoder_mask, dataset.type_dict['<SOS>'])

For Transformer.infer and Transformer.forward, the returned values diff by a torch.log operation

in infer_one(), we have

prob_t = self.predictor(decoder_step[:, -1])
return self.activation(prob_t)

However, in forward

prediction = self.predictor(decoder_output.decoder_input)
return torch.log(self.activation(prediction))

Therefore, the batch loss in both train_epoch and eval_epoch is computed by

batch_loss = criterion(batch_p[:, :-1].permute(0, 2, 1), batch_y[:, 1:])

may not be comparable for train_epoch and eval_epoch?