ImportError: cannot import name 'prepare_model_for_training' from 'peft' about peft HOT 18 CLOSED

Can confirm installing from transformers main works now.

from peft.

Hi @jordancole21
Thanks for the issue, you should use prepare_model_for_int8_training instead, the examples have been updated accordingly. Also make sure to use the main branch of peft
Thanks!

from peft.

Hi @jordancole21 Thanks for the issue, you should use prepare_model_for_int8_training instead, the examples have been updated accordingly. Also make sure to use the main branch of peft Thanks!

Awesome, thank you so much!

from peft.

It fails at get_peft_model():

So add this after:

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q", "v"],
    lora_dropout=0.05,
    bias="none",
    task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)

Will get a stacktrace soon (having too much fun finetuning it :) )

from peft.

Thanks a lot for narrowing down the issue, this should be fixed in huggingface/transformers#21688 ! You can directly use it with pip install git+https://github.com/younesbelkada/transformers.git@fix-int8-conversion

Oh perfect, thank you so much!

from peft.

It is on the main branch now, you can directly use transformers main branch and issue should be solved, feel free to close the issue if you think that it has been solved! ;)

from peft.

Thanks everyone for your detailed guidance!

from peft.

The output_embedding_layer_name argument shouldn't be necessary anymore.

from peft.

Hi @jordancole21 Thanks for the issue, you should use prepare_model_for_int8_training instead, the examples have been updated accordingly. Also make sure to use the main branch of peft Thanks!

Actually, sorry looks like I got this error on the latest version of the Flan-T5 notebook after running the 'Load your PeftModel ' part of the notebook:

TypeError: init() got an unexpected keyword argument 'has_fp16_weights'

from peft.

I am now seeing the TypeError: init() got an unexpected keyword argument 'has_fp16_weights' issue as well, even on an earlier peft commit which is weird.

It is likely something broke upstream in transformers and is unrelated, as installing the transformers the default way and not from main no longer causes this error, so may have to use an earlier transformers commit. This one from a week ago works.

huggingface/transformers@75a208e

from peft.

Hi!
Thanks for the report, I couldn't reproduce with the current main branch of transformers and peft. Here is the script I used:

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = 'google/flan-t5-large'

model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.float16, load_in_8bit=True, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

from peft import prepare_model_for_int8_training

model = prepare_model_for_int8_training(model)

Can you share with us a handy script so that I can try to reproduce on my side too?
Also can you please share the full traceback?
Thanks!

from peft.

Full trace:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-4-6b38379a6292>](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in <module>
     26 
     27 
---> 28 model = get_peft_model(model, lora_config)
     29 print_trainable_parameters(model)

5 frames
[/usr/local/lib/python3.8/dist-packages/peft/mapping.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in get_peft_model(model, peft_config)
    143     else:
    144         peft_config = _prepare_prompt_learning_config(peft_config, model_config)
--> 145     return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config)

[/usr/local/lib/python3.8/dist-packages/peft/peft_model.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, model, peft_config)
    629 
    630     def __init__(self, model, peft_config: PeftConfig):
--> 631         super().__init__(model, peft_config)
    632         self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
    633         self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation

[/usr/local/lib/python3.8/dist-packages/peft/peft_model.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, model, peft_config)
     78             self._setup_prompt_encoder()
     79         else:
---> 80             self.base_model = LoraModel(peft_config, model)
     81         if getattr(self.peft_config, "modules_to_save", None) is not None:
     82             self.modules_to_save = self.peft_config.modules_to_save

[/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, config, model)
    116         self.peft_config = config
    117         self.model = model
--> 118         self._find_and_replace()
    119         mark_only_lora_as_trainable(self.model, self.peft_config.bias)
    120         self.forward = self.model.forward

[/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in _find_and_replace(self)
    157                     new_module = Linear8bitLt(target.in_features, target.out_features, bias=bias, **kwargs)
    158                 elif isinstance(target, torch.nn.Linear) and self.peft_config.enable_lora is None:
--> 159                     new_module = Linear(target.in_features, target.out_features, bias=bias, **kwargs)
    160                 elif self.peft_config.enable_lora is not None:
    161                     kwargs.update({"enable_lora": self.peft_config.enable_lora})

[/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, in_features, out_features, r, lora_alpha, lora_dropout, fan_in_fan_out, merge_weights, **kwargs)
    285         **kwargs,
    286     ):
--> 287         nn.Linear.__init__(self, in_features, out_features, **kwargs)
    288         LoraLayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, merge_weights=merge_weights)
    289 

TypeError: __init__() got an unexpected keyword argument 'has_fp16_weights'

from peft.

Hi! Thanks for the report, I couldn't reproduce with the current main branch of transformers and peft. Here is the script I used:

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_name = 'google/flan-t5-large'

model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.float16, load_in_8bit=True, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)

from peft import prepare_model_for_int8_training

model = prepare_model_for_int8_training(model)

Can you share with us a handy script so that I can try to reproduce on my side too? Also can you please share the full traceback? Thanks!

Yeah for sure! This is the part of the code that's giving me issues. I'm running it in Google Colab on a A100 40GB.

!pip install -q bitsandbytes datasets accelerate loralib
!pip install -q git+https://github.com/huggingface/transformers.git@main git+https://github.com/huggingface/peft.git@main

from peft import prepare_model_for_int8_training

model = prepare_model_for_int8_training(model)

from peft import LoraConfig, get_peft_model, TaskType


def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=16, lora_alpha=32, 
    target_modules=["q", "v"], 
    lora_dropout=0.05, 
    bias="none", 
    task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)

from peft.

Thanks a lot for narrowing down the issue, this should be fixed in huggingface/transformers#21688 !
You can directly use it with pip install git+https://github.com/younesbelkada/transformers.git@fix-int8-conversion

from peft.

Hi guys,
I also encountered the following error after running my code:
from peft import prepare_model_for_training
ImportError: cannot import name 'prepare_model_for_training' from 'peft' (/***/***/anaconda3/lib/python3.10/site-packages/peft/__init__.py)
I installed peft as below:
pip install -q git+https://github.com/huggingface/transformers.git@main git+https://github.com/huggingface/peft.git@main
My code:

`feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path, language=language, task=task)
processor = WhisperProcessor.from_pretrained(model_name_or_path, language=language, task=task)
model = WhisperForConditionalGeneration.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto")

model.config.forced_decoder_ids = None
model.config.suppress_tokens = []

from peft import prepare_model_for_training

model = prepare_model_for_training(model, output_embedding_layer_name="proj_out")
from peft import LoraConfig, PeftModel, LoraModel, LoraConfig, get_peft_model

config = LoraConfig(r=32, lora_alpha=64, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, bias="none")

model = get_peft_model(model, config)
model.print_trainable_parameters()
`

from peft.

Could you please try prepare_model_for_kbit_training instead of prepare_model_for_training?

from peft.

I get the following error for prepare_model_for_kbit_training:

TypeError: prepare_model_for_kbit_training() got an unexpected keyword argument 'output_embedding_layer_name'

from peft.

@BenjaminBossan Dear
Thank you for your guidance!

from peft.