[BUG]I found that the parameters of model will be fully transferred to the VRAM of each process. Is this abnormal in my understanding? about deepspeed HOT 5 OPEN

@bkuster0 Thank you very much for your information. I have read it in detail. The fine-tuning of the CogVLM2 model does not support Zero3. But I still want to know what is the difference between the model supporting Zero3 and not supporting Zero3, or how should I modify the model to make it support Zero3. I hope someone can give me an answer to this question. I would be very grateful.

from deepspeed.

@tiandazhao, if using the HF+DeepSpeed integration properly then zero stage 3 would be detected at module initialization so that parameters are sharded across devices: https://huggingface.co/docs/transformers/main/en/deepspeed#non-trainer-deepspeed-integration

So, your observation is surprising. Are you able to share the log?

from deepspeed.

@tiandazhao, if using the HF+DeepSpeed integration properly then zero stage 3 would be detected at module initialization so that parameters are sharded across devices: https://huggingface.co/docs/transformers/main/en/deepspeed#non-trainer-deepspeed-integration
So, your observation is surprising. Are you able to share the log?
Since the logger generated in the process is quite messy, I will reorganize my questions.
Add Deepspeed's estimate of video memory usage: stage2

Add Deepspeed's estimate of video memory usage: stage3

When Transformers.trainer is initialized，Will enter the training loop and execute the self.accelerator.prepare method

Next, the deepspeed will initillize

Next, will create DeepSpeedEngine

then ,The _configure_distributed_model method in DeepSpeedEngine.init() will be run

In this method, it will check is_zero_init_model and dont_change_device

At this point the model has not been processed, there will be no ds_id in the parameters, and it is obvious that is_zero_init_model will definitely be False, so this is the problem. Next, self.module.to(self.device) will be executed, and then the entire model will be filled into the graphics card.

I am using three cards for training

Finally, when the training process starts, each graphics card has already stored the entire model, so it is not enough to support further training. So, the graphics memory overflows.

The above is my unprofessional and superficial understanding of the Deepspeed usage process. If there are any errors, please point them out. I also hope that you can answer my questions. Thank you very much.

from deepspeed.

Entirely same error when I fine tuning llama2. The is_zero_init_model is always False. And all params will load on each gpu without partition. @loadams @tjruwase @deepcharm

from deepspeed.

And I also confused why from_pretrained just load params to CPU memory and params load CPU to GPU memory in trainer.train.

from deepspeed.