Clarification needed on Adapter Heads in PEFT about peft HOT 2 OPEN

First a clarifying question: When you speak of adapter heads, do you mean that you added them when training the model by setting LoraConfig(modules_to_save=...)? Note that if you did not set it explicitly but used a preconfigured model, it might have been set automatically (check the adapter_config.json).

2. When merging multiple adapters using add_weighted_adapter, I would expect the corresponding adapter heads to be merged as well so that the merged adapter can handle the combined tasks of the individual adapters.

This is definitely not done. In fact, there is no trivial way to merge adapter heads in a way that would preserve their functionality.

from peft.