Ignore keys for modules to save about peft HOT 1 OPEN

I'm not quite sure what you mean. On the one hand, you mention targeting LoRA layers, which implies you want to exclude certain layers from target_modules. This is already possible as target_modules accepts a regex, which is flexible enough to cover most use cases. As an example, for the OPT model, if I pass LoraConfig(target_modules=r".*\.(?!5\b)\d+\.self_attn\.k_proj"), I'll match all k_proj layers except for layer 5.

On the other hand, you mention modules_to_save in the title. Note that this is not related to applying LoRA, the modules listed there are fully fine-tuned and saved in the checkpoint file. This should not automatically include your extra attention layers.

from peft.