Comments (18)
Can confirm installing from transformers
main
works now.
from peft.
Hi @jordancole21
Thanks for the issue, you should use prepare_model_for_int8_training
instead, the examples have been updated accordingly. Also make sure to use the main
branch of peft
Thanks!
from peft.
Hi @jordancole21 Thanks for the issue, you should use
prepare_model_for_int8_training
instead, the examples have been updated accordingly. Also make sure to use themain
branch ofpeft
Thanks!
Awesome, thank you so much!
from peft.
It fails at get_peft_model()
:
So add this after:
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q", "v"],
lora_dropout=0.05,
bias="none",
task_type="SEQ_2_SEQ_LM"
)
model = get_peft_model(model, lora_config)
Will get a stacktrace soon (having too much fun finetuning it :) )
from peft.
Thanks a lot for narrowing down the issue, this should be fixed in huggingface/transformers#21688 ! You can directly use it with
pip install git+https://github.com/younesbelkada/transformers.git@fix-int8-conversion
Oh perfect, thank you so much!
from peft.
It is on the main
branch now, you can directly use transformers
main
branch and issue should be solved, feel free to close the issue if you think that it has been solved! ;)
from peft.
Thanks everyone for your detailed guidance!
from peft.
The output_embedding_layer_name
argument shouldn't be necessary anymore.
from peft.
Hi @jordancole21 Thanks for the issue, you should use
prepare_model_for_int8_training
instead, the examples have been updated accordingly. Also make sure to use themain
branch ofpeft
Thanks!
Actually, sorry looks like I got this error on the latest version of the Flan-T5 notebook after running the 'Load your PeftModel
' part of the notebook:
TypeError: init() got an unexpected keyword argument 'has_fp16_weights'
from peft.
I am now seeing the TypeError: init() got an unexpected keyword argument 'has_fp16_weights'
issue as well, even on an earlier peft
commit which is weird.
It is likely something broke upstream in transformers
and is unrelated, as installing the transformers
the default way and not from main
no longer causes this error, so may have to use an earlier transformers
commit. This one from a week ago works.
huggingface/transformers@75a208e
from peft.
Hi!
Thanks for the report, I couldn't reproduce with the current main
branch of transformers
and peft
. Here is the script I used:
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_name = 'google/flan-t5-large'
model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.float16, load_in_8bit=True, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
Can you share with us a handy script so that I can try to reproduce on my side too?
Also can you please share the full traceback?
Thanks!
from peft.
Full trace:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
[<ipython-input-4-6b38379a6292>](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in <module>
26
27
---> 28 model = get_peft_model(model, lora_config)
29 print_trainable_parameters(model)
5 frames
[/usr/local/lib/python3.8/dist-packages/peft/mapping.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in get_peft_model(model, peft_config)
143 else:
144 peft_config = _prepare_prompt_learning_config(peft_config, model_config)
--> 145 return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config)
[/usr/local/lib/python3.8/dist-packages/peft/peft_model.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, model, peft_config)
629
630 def __init__(self, model, peft_config: PeftConfig):
--> 631 super().__init__(model, peft_config)
632 self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
633 self.base_model.prepare_inputs_for_generation = self.prepare_inputs_for_generation
[/usr/local/lib/python3.8/dist-packages/peft/peft_model.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, model, peft_config)
78 self._setup_prompt_encoder()
79 else:
---> 80 self.base_model = LoraModel(peft_config, model)
81 if getattr(self.peft_config, "modules_to_save", None) is not None:
82 self.modules_to_save = self.peft_config.modules_to_save
[/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, config, model)
116 self.peft_config = config
117 self.model = model
--> 118 self._find_and_replace()
119 mark_only_lora_as_trainable(self.model, self.peft_config.bias)
120 self.forward = self.model.forward
[/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in _find_and_replace(self)
157 new_module = Linear8bitLt(target.in_features, target.out_features, bias=bias, **kwargs)
158 elif isinstance(target, torch.nn.Linear) and self.peft_config.enable_lora is None:
--> 159 new_module = Linear(target.in_features, target.out_features, bias=bias, **kwargs)
160 elif self.peft_config.enable_lora is not None:
161 kwargs.update({"enable_lora": self.peft_config.enable_lora})
[/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py](https://lu1as38218r-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab-20230216-060056-RC01_510471064#) in __init__(self, in_features, out_features, r, lora_alpha, lora_dropout, fan_in_fan_out, merge_weights, **kwargs)
285 **kwargs,
286 ):
--> 287 nn.Linear.__init__(self, in_features, out_features, **kwargs)
288 LoraLayer.__init__(self, r=r, lora_alpha=lora_alpha, lora_dropout=lora_dropout, merge_weights=merge_weights)
289
TypeError: __init__() got an unexpected keyword argument 'has_fp16_weights'
from peft.
Hi! Thanks for the report, I couldn't reproduce with the current
main
branch oftransformers
andpeft
. Here is the script I used:import torch from transformers import AutoModelForSeq2SeqLM, AutoTokenizer model_name = 'google/flan-t5-large' model = AutoModelForSeq2SeqLM.from_pretrained(model_name, torch_dtype=torch.float16, load_in_8bit=True, device_map="auto") tokenizer = AutoTokenizer.from_pretrained(model_name) from peft import prepare_model_for_int8_training model = prepare_model_for_int8_training(model)Can you share with us a handy script so that I can try to reproduce on my side too? Also can you please share the full traceback? Thanks!
Yeah for sure! This is the part of the code that's giving me issues. I'm running it in Google Colab on a A100 40GB.
!pip install -q bitsandbytes datasets accelerate loralib
!pip install -q git+https://github.com/huggingface/transformers.git@main git+https://github.com/huggingface/peft.git@main
from peft import prepare_model_for_int8_training
model = prepare_model_for_int8_training(model)
from peft import LoraConfig, get_peft_model, TaskType
def print_trainable_parameters(model):
"""
Prints the number of trainable parameters in the model.
"""
trainable_params = 0
all_param = 0
for _, param in model.named_parameters():
all_param += param.numel()
if param.requires_grad:
trainable_params += param.numel()
print(
f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
)
lora_config = LoraConfig(
r=16, lora_alpha=32,
target_modules=["q", "v"],
lora_dropout=0.05,
bias="none",
task_type="SEQ_2_SEQ_LM"
)
model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
from peft.
Thanks a lot for narrowing down the issue, this should be fixed in huggingface/transformers#21688 !
You can directly use it with pip install git+https://github.com/younesbelkada/transformers.git@fix-int8-conversion
from peft.
Hi guys,
I also encountered the following error after running my code:
from peft import prepare_model_for_training
ImportError: cannot import name 'prepare_model_for_training' from 'peft' (/***/***/anaconda3/lib/python3.10/site-packages/peft/__init__.py)
I installed peft as below:
pip install -q git+https://github.com/huggingface/transformers.git@main git+https://github.com/huggingface/peft.git@main
My code:
`feature_extractor = WhisperFeatureExtractor.from_pretrained(model_name_or_path)
tokenizer = WhisperTokenizer.from_pretrained(model_name_or_path, language=language, task=task)
processor = WhisperProcessor.from_pretrained(model_name_or_path, language=language, task=task)
model = WhisperForConditionalGeneration.from_pretrained(model_name_or_path, load_in_8bit=True, device_map="auto")
model.config.forced_decoder_ids = None
model.config.suppress_tokens = []
from peft import prepare_model_for_training
model = prepare_model_for_training(model, output_embedding_layer_name="proj_out")
from peft import LoraConfig, PeftModel, LoraModel, LoraConfig, get_peft_model
config = LoraConfig(r=32, lora_alpha=64, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, bias="none")
model = get_peft_model(model, config)
model.print_trainable_parameters()
`
from peft.
Could you please try prepare_model_for_kbit_training
instead of prepare_model_for_training
?
from peft.
I get the following error for prepare_model_for_kbit_training
:
TypeError: prepare_model_for_kbit_training() got an unexpected keyword argument 'output_embedding_layer_name'
from peft.
@BenjaminBossan Dear
Thank you for your guidance!
from peft.
Related Issues (20)
- RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn HOT 1
- LoftQ init without scaling HOT 1
- Gumbel-sigmoid (poly) HOT 2
- DoRA with fsdp model cannot work HOT 14
- TypeError: WhisperForConditionalGeneration.forward() got an unexpected keyword argument 'input_ids' HOT 4
- Linear4bit not supported error when running QDoRA_finetuning.ipynb cookbook HOT 1
- Support optimum-quanto
- Lora initialisation with olora and pissa not working with quantisation. HOT 7
- Issue with PEFT model save_pretrained HOT 5
- Missing PEFT model HOT 2
- Incorporating contrastive prefixes with PrefixTuning HOT 1
- QLora with DeepSpeed support HOT 5
- XLoRA: training issues, Gradients will be None HOT 7
- Cannot apply both PEFT QLoRA and DeepSpeed ZeRO3 HOT 7
- With tied embeddings adapter merged to tied layers HOT 3
- replace_lora_weights_loftq() throw HFValidationError when base model is loaded using a local path HOT 2
- Unable to load peft checkpoint HOT 7
- If you are interested in multimodal large models, you can join our group
- Unobvious behavior of model after applying merge_and_unload HOT 3
- Cannot use prefix tuning on quantized Codellama HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from peft.