[BUG] the gpt model cannot run in specified container about megatron-deepspeed HOT 4 CLOSED

the following is detailed error message:
Traceback (most recent call last):
File "pretrain_gpt.py", line 276, in
pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
File "/root/large_model/Megatron-DeepSpeed/megatron/training.py", line 163, in pretrain
iteration = train(forward_step_func,
File "/root/large_model/Megatron-DeepSpeed/megatron/training.py", line 877, in train
train_step(forward_step_func,
File "/root/large_model/Megatron-DeepSpeed/megatron/training.py", line 499, in train_step
losses_reduced = forward_backward_func(
File "/root/large_model/Megatron-DeepSpeed/megatron/schedules.py", line 154, in forward_backward_no_pipelining
backward_step(optimizer, input_tensor, output_tensor,
File "/root/large_model/Megatron-DeepSpeed/megatron/schedules.py", line 104, in backward_step
torch.autograd.backward(output_tensor, grad_tensors=output_tensor_grad)
File "/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py", line 145, in backward
Variable._execution_engine.run_backward(
File "/opt/conda/lib/python3.8/site-packages/torch/autograd/function.py", line 89, in apply
return self._forward_cls.backward(self, *args) # type: ignore
File "/root/large_model/Megatron-DeepSpeed/megatron/mpu/random.py", line 311, in backward
torch.autograd.backward(outputs, args)
File "/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py", line 145, in backward
Variable._execution_engine.run_backward(
RuntimeError: element 1 of tensors does not require grad and does not have a grad_fn

from megatron-deepspeed.

I found the same problem.
help！

from megatron-deepspeed.

git checkout v2.4

from megatron-deepspeed.

git checkout v2.4

thanks, i have trained gpt by megatron-lm repo successfully.
i will try this way later.

from megatron-deepspeed.