CUDA out of memory about pixelssl HOT 4 CLOSED

Thank you very much for your answer.

I used 4 GPUs (each 12G). During the gct experiment, I found that the main GPU used for integration would exceed 12G memory. Next, i will try to solve the imbalance problem of memory by using DistributedDataParallel.

from pixelssl.

Hi, thanks for your attention.

Which GPU are you using? For PSPNet, if you use 4 GPUs for training, the GCT algorithm requires about 12G memory (each GPU). I have tried to optimize the algorithm, but it still requires a lot of memory during training.
If you change the im_size to 321, the performance may be decreased.

Is it possible for you to use 8 GPUs for training? If so, in the script pspnet_pascalvoc_1-8_sslgct.py, you can change:

('lr', 0.00025),
('batch_size', 4),
('unlabeled_batch_size', 2), 

to

('lr', 0.000125),
('batch_size', 2),
('unlabeled_batch_size', 1), 

You can also use the results reported by us. For PSPNet + GCT under 1/2 label, the SSL performance is 73.92. This results will be uploaded soon.

from pixelssl.

Yes. If you use DistributedDataParallel, the memory will be balanced better. I am sorry that I do not have time to implement it recently.

from pixelssl.

Thank you again.

from pixelssl.