Inferencing with adapter vs Inferencing with fused model about mlx-examples HOT 2 OPEN

Hi awni,

Thank you for your response :)

• Actually I fine-tuned the model after converting it with the -q flag so I guess it's a quantized model, hence QLoRA.
• I can say that the fused model is worse than the original phi-2.

I am not getting any particular error, it's just that the output isn't exactly what I would expect.
What I am going to set up is a set of prompts that I can use to compare the base and the fine-tuned model. I will surely let you know.

from mlx-examples.

Hmm, some questions:

• Was it a quantized model that you fine-tuned or an fp16 model?
• Is the fused model even worse than the original baseline phi-2?

Sometimes fusing can reduce precision since you are adding a small update to the base model. That's especially the case when you fuse into a quantized model.

But I am not certain that is the issue here. It would be useful to have a little more information to help debug.

from mlx-examples.