[BUG] Zero3 causes AttributeError: 'NoneType' object has no attribute 'numel' in continual training about deepspeed HOT 3 OPEN

I have the same issue and solved it by just reloading the whole model on every iteration.
Though now I am getting oom due to the memory not freeing after the second epoch, but I guess that is a different issue.

from deepspeed.

@Xirid, please try the following API to free engine memory
https://deepspeed.readthedocs.io/en/latest/zero3.html#gpu-memory-management

from deepspeed.

我也遇到了在 for 循环中创建新的数据集和新的训练器，然后调用trainer.train()。
在 for 循环的第一次迭代中，训练工作正常。然而，在第二次迭代中，报错为错误： AttributeError: 'NoneType' object has no attribute 'numel'
尝试释放GPU，但没有用，释放GPU如下：# Free GPU memory consumed by model parameters
ds_engine.empty_partition_cache()

from deepspeed.