rate_num_examples Doesn't work with new tasks about text-to-text-transfer-transformer HOT 6 CLOSED

This is a known issue that we are working on. For now, you can hardcode the number of examples in each task when constructing the mixture.

from text-to-text-transfer-transformer.

@craffel Thanks for the reply.

For now, I set it manually to:

t5.data.MixtureRegistry.add(
"mixtask",
[["task1",0.34], ["task2",0.34],["task3",0.21],["task3",0.11]],
)

Hopefully, you fix it soon.

from text-to-text-transfer-transformer.

Thanks, yes, it is at the top of our to-do list!

from text-to-text-transfer-transformer.

@craffel Is the rate is correct as in the above example or should we use the actual number of samples?

I have checked the code and the task sampling is generated using "tf.data.experimental.sample_from_datasets" which uses "math_ops.log".

I am not sure should we pass the actual number of examples for each task or the rate of each task which is sum of 1 ?

using rates the math_ops.log leads to :
[0.34, 0.34, 0.21, 0.11] -> [-1.0788096 -1.0788096 -1.5606477 -2.207275 ]
using actual number of samples :
[10791.0, 10791.0, 6621.0, 3744.0] -> [9.286468 9.286468 8.798001 8.22791 ]

each one of them leads to totally different results using the multi-task training.

from text-to-text-transfer-transformer.

It uses a log to convert the weights to logits before feeding them into a multinomial distribution for sampling. This is not a detail you need to worry about. It will sample proportionally according to the weights as you'd expect.

from text-to-text-transfer-transformer.

test

from text-to-text-transfer-transformer.