Comments (7)
Hello LoRA Model adds modules to original model in place. Please use below function for knowing the number of trainable param percentage:
def print_trainable_parameters(model):
"""
Prints the number of trainable parameters in the model.
"""
trainable_params = 0
all_param = 0
for _, param in model.named_parameters():
all_param += param.numel()
if param.requires_grad:
trainable_params += param.numel()
print(
f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
)
from peft.
Thanks!
from peft.
If we do print_trainable_parameters(model)
and print_trainable_parameters(lora_model)
, it prints the same numbers:
trainable params: 667493 || all params: 86466149 || trainable%: 0.7719703117574949
trainable params: 667493 || all params: 86466149 || trainable%: 0.7719703117574949
Reference Colab: https://colab.research.google.com/drive/1GtKXiVyINz2rRnd5FSMh2WnTjt7nOTEn?usp=sharing#scrollTo=-4zp34QGjEoi
I know the LoRAModel
adds modules in place but this situation might confuse users that are interested in investigating parameter efficiency gains from LoRA in terms of the trainable parameters. Could we provide some kind of a wrapper that would remove the in-place modifications?
from peft.
Hello, the number of trainable parameters is just 0.77% of the total number of params in the model, which should indicate the parameter efficiency in fine-tuning. I am confused about the issue here. In the usual case for causal language modelling or seq2seq, only Lora params are trainable and all the base model params are frozen; in contrast to the case of full-finetuning wherein all the base model params are trainable.
from peft.
Sorry if I wasn't clear.
In the above issue model
is the original model and lora_model
is the LoRA modified version of model
. My issue is that the total number of trainable parameters of the original model should be as is and that is not the case here (probably because of the in-place modifications introduced by PEFT).
Let's illustrate what I am saying with an example:
from transformers import AutoModelForImageClassification
model_checkpoint = "google/vit-base-patch32-224-in21k"
label2id = {"dog": 0, "cat": 1, "mouse": 2}
id2label = {v: k for k, v in label2id.items()}
model = AutoModelForImageClassification.from_pretrained(
model_checkpoint,
label2id=label2id,
id2label=id2label,
ignore_mismatched_sizes=True,
)
def print_trainable_parameters(model):
"""
Prints the number of trainable parameters in the model.
"""
trainable_params = 0
all_param = 0
for _, param in model.named_parameters():
all_param += param.numel()
if param.requires_grad:
trainable_params += param.numel()
print(
f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
)
print_trainable_parameters(model)
As expected, it prints
trainable params: 87457539 || all params: 87457539 || trainable%: 100.0
After lora modifications are added we see the reduction in the number of total trainable parameters.
lora_model = LoraModel(config, model)
print_trainable_parameters(lora_model)
However, now when I call print_trainable_parameters(model)
(notice that it's model
and NOT lora_model
) I get what I got when calling print_trainable_parameters(lora_model)
.
While that is expected since the LoRA modifications are done din-place as per #41 (comment), this might be confusing for users as I mentioned in #41 (comment).
Let me know what I am trying to convey is still unclear.
from peft.
Hello, LoRA being modifying model inplace makes it widely applicable. On the other hand, I don't think this could be an issue if you compare the trainable params before and after creating PEFt model as shown below:
from peft.
Cool cool! Just want to ensure that the users are aware of the in-place modification :)
from peft.
Related Issues (20)
- AttributeError: 'NoneType' object has no attribute 'shape' HOT 14
- Merging adapters for BF16 model on CPU is very slow HOT 1
- LoraConfig.__init__() got an unexpected keyword argument 'layer_replication' when using custom model handler HOT 2
- RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn HOT 1
- LoftQ init without scaling HOT 1
- Gumbel-sigmoid (poly) HOT 2
- DoRA with fsdp model cannot work HOT 14
- TypeError: WhisperForConditionalGeneration.forward() got an unexpected keyword argument 'input_ids' HOT 4
- Linear4bit not supported error when running QDoRA_finetuning.ipynb cookbook HOT 1
- Support optimum-quanto
- Lora initialisation with olora and pissa not working with quantisation. HOT 7
- Issue with PEFT model save_pretrained HOT 5
- Missing PEFT model HOT 2
- Incorporating contrastive prefixes with PrefixTuning HOT 1
- QLora with DeepSpeed support HOT 5
- XLoRA: training issues, Gradients will be None HOT 7
- Cannot apply both PEFT QLoRA and DeepSpeed ZeRO3 HOT 7
- With tied embeddings adapter merged to tied layers HOT 3
- replace_lora_weights_loftq() throw HFValidationError when base model is loaded using a local path HOT 2
- Unable to load peft checkpoint HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from peft.