训练一直卡在loading base model中, 使用deepspeed zero3.config的时候稳定复现 about monkey HOT 12 OPEN

same question.

from monkey.

same question.

请问你解决了么？

from monkey.

No. I haven't solved it yet. Actually, I gave up🥲 It's difficult to use the translator. Please speak in English...

from monkey.

附内存占用情况

from monkey.

There are two types of a100: 40GB and 80GB. The author seems to have trained with 80GB.

from monkey.

When I use zero3, there is a situation where the loss is larger than when I use zero2. It seems that the model did not initialize successfully. Have you ever encountered this when using zero3?

The first picture shows the loss of zero3 and the second picture shows the loss of zero2, and there is a parameter mismatch issue during model initialization, as follows:

from monkey.

Training has been stuck at loading the base model, and I haven't been able to successfully train yet.

from monkey.

Does it training from scratch or loading the textmoney weights finetune?

from monkey.

Does it training from scratch or loading the textmoney weights finetune?
loading the monkey weights finetune

from monkey.

This is not good, we need train from scratch

from monkey.

Could you please give more detail information about your training? Our model use the Qwen-VL as pretrained model and it can work well in 8xA800 80G with ZeRO2.

from monkey.

Could you please give more detail information about your training? Our model use the Qwen-VL as pretrained model and it can work well in 8xA800 80G with ZeRO2.您能提供有关您的培训的更多详细信息吗？我们的模型使用 Qwen-VL 作为预训练模型，它可以在 8xA800 80G 和 ZeRO2 中很好地工作。

40G 8卡 A100

from monkey.