NaNs in sequence classifier output about aqlm HOT 2 OPEN

Hi!

As of right now, we only support quantized models with LM head (i.e. AutoModelForCausalLM).
For classifier, my guess is that the model created an empty classification head but it did not have any (quantized) weights associated with it. When initialized with empty data, quantized layers have undefined behavior.

Your workaround with explicitly initializing classifier head looks like a good (perhaps best) solution to this problem.

Unfortunately, we currently do not have time to explicitly support non-LM models (i.e. token classification, sequence classification, qa, etc). If you have bandwidth to implement such support without breaking LM models, we would welcome this as a pull request.

from aqlm.

This issue is stale because it has been open for 30 days with no activity.

from aqlm.