Problem with latency about peft HOT 5 CLOSED

I've yet to use the library - just looked at the documentation. Could be something along the lines of:

peft_config = LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM, inference_mode=True, r=8, lora_alpha=32, lora_dropout=0.1
)

model = ORTModelForSeq2SeqLM.from_pretrained(model_name_or_path)
model = get_peft_model(model, peft_config)

Note the inference mode=True. If I read the paper right I believe this should calculate W' = W0 + BA -- updating the weights such that it behaves as a normal transformer of size W0.

from peft.

Hello @PetroMaslov, For using ONNX with PEFT LoRA model, please refer to the example in the PR #118. Let us know if that solves the issue.

from peft.

@pacman100
great, thank you very much.
I checked that for me onnx (flan-t5-large) is faster than deepspeed. Does it so or I do something wrong?

from peft.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

from peft.

@PetroMaslov after converting the LoRA finetuned T5-large model to ONNX, did you see degraded model outputs? Using a bigscience/bloom base model, I can perform inference after exporting to ONNX, but the model predictions become nonsensical 🤔 #670

from peft.