PyTorch Distributed Training about fixmatch-pytorch HOT 1 CLOSED

Regarding

1. Why don't you reduce losses across GPU ...

You don't need to and should not reduce loss manually. When you call loss.backward() the gradient is reduced automatically.

As described in this answer

on loss.backward() all processes reduce their gradients. As loss.backward() returns, the gradients of your model parameters will be the same, and the optimizer in each process will perform the exact same update to the model parameters

2. When I run the code on single GPU and multiple GPUs ...

Distributed training does not always make training faster. In my opinion, the point of using distributed training is to have a larger effective batch size that is otherwise too large for a single GPU. It is normal that when you have a small batch size (i.e. 16 per GPU in your case), distributed is slower than a single GPU since there are communication overheads between different processes.

Also, distributed training could degrade performance. See this answer for detail. You might need to use a different learning rate.

from fixmatch-pytorch.