More experimental information on C4 dataset about text-to-text-transfer-transformer HOT 4 CLOSED

Hi, all of the results in the table had the same amount of training time and were trained on the same hardware. Note that the model was pre-trained on the same number of tokens for each of those experiments, the dataset itself was just artificially limited in size so that it was repeated over the course of pre-training. The training time will scale with the TPU size up to about a v3-32, at which point the model becomes input bound.

from text-to-text-transfer-transformer.

Hi, all of the results in the table had the same amount of training time and were trained on the same hardware. Note that the model was pre-trained on the same number of tokens for each of those experiments, the dataset itself was just artificially limited in size so that it was repeated over the course of pre-training. The training time will scale with the TPU size up to about a v3-32, at which point the model becomes input bound.

Can you provide the details about training time and hardware？So we can evaluate the pre-training cost. @craffel

from text-to-text-transfer-transformer.

The training time/cost for those experiments is the same as the baseline experimental setup. Most of our experiments were run on TPU v3-8 accelerators (referred to simply as a "Cloud TPU v3" in the Cloud docs).

from text-to-text-transfer-transformer.

Get it!

from text-to-text-transfer-transformer.