Questions about the implementation details of batched_forward_pass and respond_to_batch about trl HOT 1 CLOSED

1. The model outputs predictions for the next token whereas the log_probs are the log probabilities for the current token. This simply aligns the two.

2. The main motivation was to decouple the generation from the training as much as possible. Since it takes a fraction of the time of the backward pass the speedup would be minimal. That way the PPOTrainer interface is cleaner.

3. That's possible. It could be that the transformer function generate handles this, but I had to implement my own, simple decoding function since the model would exploit several aspects of it. See the comments here about the custom response function. Feel free to make a PR if you can fix the weaknesses and improve the performance.

Cheers,
Leandro

from trl.