thanks to you share,
i want to konw the model btez-fb-large is train by what
also use in this model or other things
if also train by the model, what the difference between them

Model too slow with XLA TPU
any suggestions please @abhishekkrthakur

def _mp_fn(index):
        device = xm.xla_device()
        # We wrap this 
        model = WRAPPED_MODEL.to(device)
        print('call fit')
        model.fit(
        train_dataset,
        train_bs=args.batch_size,
        device=device,
        epochs=args.epochs,
        callbacks=[es],
        fp16=False,
        accumulation_steps=args.accumulation_steps,
        )
        #train_nli(model)
    print('x')
    xmp.spawn(_mp_fn, start_method="fork")

It will raise an error when I tried to train tez model on multiple gpus.
model = nn.DataParallel(model, device_ids=[0,1,2,3])
AttributeError: 'DataParallel' object has no attribute 'fit'

How can I do this? Thank you so much

Hi ya,

Thanks for your great repo. Can I ask something about the forward method of the model?

In the forward method, I can see that you are passing the transformer output through the classification head with 5 different dropout layer and averaging the logits & the loss to get the final output. Can I ask why?

I also tried using only 1 dropout without these auxiliary loss but the result is substantially worse. Can I ask if you have any idea why this is the case?

Thank you