[Discussion] mp: duplicate of torch.cuda.set_device(local_rank) and images = images.cuda(local_rank, non_blocking=True) about pytorch-distributed HOT 4 OPEN

Thank you!
I mean when I want to terminate the training, I would use Ctrl+D in the terminal. Sometimes one process would still be there occupying the GPU. Then I have to kill the process by its ID.

from pytorch-distributed.

Of course, if you do torch.cuda.set_device(local_rank) it is okay to use model.cuda() instead of model.cuda(args.gpu). The official repo kept the old fashion of best practice based on previous versions.

And what do u mean by 'cannot kill all the processes by Ctrl+D'? Everything goes well for me.

from pytorch-distributed.

Thank you!
I mean when I want to terminate the training, I would use Ctrl+D in the terminal. Sometimes one process would still be there occupying the GPU. Then I have to kill the process by its ID.

Me too... Do you have any advice?

from pytorch-distributed.

I finally encountered a situation where the GPU memory was not cleared after using ctrl+c to kill the process today. It happened when I used a custom CUDA module. If you encounter the same situation, maybe you can refer to facebookresearch/fairseq#487, which works for me (even though does not look elegant).

I will try to find the cause of this problem, but it seems not easy to locate...

from pytorch-distributed.