NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet about llm-shearing HOT 3 CLOSED

It seems to be a lack of support for native FSDP for these settings. Since our repo does not use transformers code, we probably cannot easily use the PR? Plus, Composer has another wrapper on top of FSDP which might add extra complexity.

from llm-shearing.

I am currently encountering this problem when running a pruning experiment on a single GPU. I want to solve it by turning off FSDP, but this code seems to be incompatible with settings without FSDP and encounters many problems. But when I turned on the FSDP settings, this problem prevented me from getting Checkpoint.
Are you able to share the general configuration of your machine, such as cuda, FSDP and other infrastructure? Will you encounter this problem when running a program on a single GPU?

from llm-shearing.

Hiii I never experimented with single GPUs. You might need to deactivate the FSDP setup in the yaml scripts to completely turn off FSDP by setting`fsdp_config=null.

• I use cuda 12.2 but I think cuda 12.0 and cuda 11.7 should work as well.
• You can find the default FSDP set up in the yaml file
• I use 8 GPUs for pruning and 16 for continued pre-training

from llm-shearing.