2080Ti多卡训练，GPU利用率忽高忽低，这个怎么回事 about pytorch-distributed HOT 2 CLOSED

As far as I know, data migration generally does not take long. But if you use a lot of GPUs, nvlink is effective. In the demos, my GPU usage was maintained at 100%. Please check if there is any synchronization operation in your program causing other GPUs to wait longer.

from pytorch-distributed.

yons 10829 0.2 0.1 2991056 134104 pts/9 S+ 19:34 0:05 python3 -m torch.distributed.launch --nproc_per_node=2 train.py -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2
yons 10882 101 4.9 31145112 6472044 pts/9 Sl+ 19:34 29:48 /usr/bin/python3 -u train.py --local_rank=0 -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2
yons 10883 101 8.2 37004736 10941320 pts/9 Sl+ 19:34 29:51 /usr/bin/python3 -u train.py --local_rank=1 -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2
yons 11082 39.5 4.1 21305204 5501024 pts/9 Dl+ 19:34 11:29 /usr/bin/python3 -u train.py --local_rank=0 -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2
yons 11091 39.4 4.1 21308788 5506644 pts/9 Dl+ 19:34 11:27 /usr/bin/python3 -u train.py --local_rank=0 -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2
yons 11101 39.6 7.5 27062080 9980424 pts/9 Dl+ 19:34 11:30 /usr/bin/python3 -u train.py --local_rank=1 -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2
yons 11110 39.6 7.5 27060044 9977892 pts/9 Dl+ 19:34 11:30 /usr/bin/python3 -u train.py --local_rank=1 -net ir -depth 50 -b 200 -w 2 -lr 0.1 -num_gpus 2

训练一段时间后，取数据进程状态变成了Dl+，gpu利用率就忽高忽低了，这是为什么啊

from pytorch-distributed.