Case Study: Instruction Tuning on AQLM Models about aqlm HOT 7 CLOSED

I've merged #39 and released aqlm==1.1.0 where I got rid of the need to use aqlm.optimize_for_training(). Everything is determined automatically from here on.

from aqlm.

Sounds great! We will instruct users to use the latest AQLM in our training framework

from aqlm.

Hi @hiyouga !
It's, sadly, not properly documented yet, but you should do:

import aqlm

with aqlm.optimize_for_training():
    model = AutoModelForCausalLM.from_pretrained(...)

The thing is, there a few ways to compute a forward pass and some of them work better for very small number of tokens (e.g. generation), and some are optimized for large batch sizes (e.g. training). We're hoping to be able to determine which kernels to use dynamically in later versions of aqlm, but, for now, please add that wrapped explicitly. Also, keep in mind, that a model loaded under that wrapper will be very slow on generation. We're working on making it a more pleasant experience!

from aqlm.

@BlackSamorez Indeed! It's very important to me, I will try to fine-tune the model again following your advice. Thanks for pointing it out!

from aqlm.

A bit more context: those are the speeds for a typical layer on an RTX 3090 GPU. We have a kernel for a single token pass (generation), which is slightly faster than fp16, and we have a kernel which introduces a huge but constant overhead over fp16, meaning it's asymptotically as fast as fp16.

As of now, we don't have a good enough kernel for anything between 4 and 4000 tokens processed in a pass. We're hoping to implement them someday.

from aqlm.

I see. The generation is much faster than training, and it might also be related to the gradient checkpointing technique in training.

from aqlm.

This issue is stale because it has been open for 30 days with no activity.

from aqlm.