text generation about text-to-text-transfer-transformer HOT 1 CLOSED

T5 is trained in a text-to-text framework where every task is cast as "generating text", e.g. generating the text corresponding to a label (sentiment analysis input "this movie was terrible" -> target "negative") or generating a transformation of some input text ("das ist gut" -> "that is good"). But I am guessing you mean more general text generation, like GPT-2 or CTRL (e.g. continuing on generating text conditioned on a prefix). Our pre-trained models are not trained to do this, none of the objectives or tasks correspond to this directly. Our "unlabeled data" objective is more like a fill-in-the-blank/denoising objective. If you want, you could fine-tune a pre-trained model on a "continue this text" objective using the prefix_lm preprocessor: https://github.com/google-research/text-to-text-transfer-transformer/blob/master/t5/data/preprocessors.py#L562

from text-to-text-transfer-transformer.