multi-gpu training fails about gearnet HOT 5 CLOSED

Hi!

The memory of our cluster is 500G, which is enough for loading EC and AF DB splits four times. This protocol follows the module-level data parallelism in Pytorch. To save memory, you can shrink the size of each split in AF DB.

from gearnet.

Hello, @Oxer11

When using 4 GPUs, it seems that the Memory will be used out at the loading data stage.

from gearnet.

Hi Shoufa!

Thanks for raising this issue! I think this is because loading the whole dataset four times will take very large memory. Here I suggest:

1. use a machine with larger cpu (240G should be enough)
2. use 2 gpus instead of 4 gpus
3. try to turn on the lazy option when loading dataset (this will avoid loading the whole dataset into cpu)

from gearnet.

Hi @Oxer11 ,

Thanks for your reply. I was wondering if it is necessary to load independent data for each process, ie, is it possible to let all processes share the loaded data?

from gearnet.

Hi, @Oxer11

How much memory does GearNet need for the AlphaFold dataset at the pertaining stage?

from gearnet.