Torchrun / torch.distributed.run throws RendezvousConnectionError / DistNetworkError (Connection reset by peer) about pytorch HOT 1 CLOSED

I want to close this ticket with a couple of comments on how I resolved a whole bunch of issues on AMD Raedon RX 7900 XTX.

1.) Issue above was because of differences of Pytorch versions. One was compiled with ROCM 5.6 while the other was compiled with ROCM 6.0 (even though ROCM itself was upgraded to 6.0 in both).

2.) Master node was throwing a SEGSEGV error which showed up on /var/log/syslog as something happening with librocm/libhip. Turns out you absolutely have to set export PYTORCH_ROCM_ARCH="gfx1100" for this SEGSEGV error to go away. Consider setting the following as well export HSA_OVERRIDE_GFX_VERSION=11.0.0. These will resolve the SEGSEGV on Master node.

3.) Worker node still had a whole bunch of NCCL (RCCL) related issues. I found out that setting the following environment variables will clear all of those.

export LOGLEVEL=DEBUG
export TORCH_CPP_LOG_LEVEL=INFO
export TORCH_DISTRIBUTED_DEBUG=INFO


export NCCL_P2P_DISABLE=0
export NCCL_IB_DISABLE=1
export NCCL_P2P_DISABLE=1
export NCCL_DEBUG=INFO

if [ "$SLURM_PROCID" == "0" ]; then
   # Change these to your actual ethernet interfaces by doing ifconfig
    export NCCL_SOCKET_IFNAME="enp74s0"
else
    export NCCL_SOCKET_IFNAME="enp5s0"
fi





export PYTORCH_ROCM_ARCH="gfx1100"
export HSA_OVERRIDE_GFX_VERSION=11.0.0
export HIP_VISIBLE_DEVICES=0

The debug flags, obviously, are to get more information inside the error logs. Also, the backend for torchrun is different from the backend for init_process_group!

You should now be able to test dist.send(), dist.recv(), and dist.barrier()! Congratulations!

from pytorch.