No training when using 2 nodes and torchrun about dora HOT 6 OPEN

Are you change the node id for the second command ?

from dora.

@adefossez Yes, sure. I did some debugging and noticed that the time it took to complete each step increased greatly. In fact, learning is happening, but it is very slow. Moreover, if I run it on one machine, the process goes quickly. I expected that with the addition of N nodes the time to complete one epoch would be reduced, but it increased greatly in the end.

from dora.

It will depend on the batch size, and whether you are specifying a per GPU or overall batch size (I recommend the later so that the XP meaning doesn't change based on how many gpus are used). What codebase is this for ?

from dora.

Also it will depend if you have good interconnect between nodes!

from dora.

Check your network config, you might have a firewall, security group, etc blocking access on the ports torchrun is using. If you get this working please update us, I'm currently going through the same headache...

from dora.

@Tristan-Kosciuch Yes, everything works for me. To be honest, I don’t remember what the problem was. I reinstalled the environment, updated all the libraries and the problem was solved.

from dora.