How model_seqlen affects quantization quality about aqlm HOT 6 CLOSED

Hi! It is recommended to use the seq_len the model you're quantizing was trained on (4096 for Llama-2, 8192 for mistral/mixtral). To reduce the number of samples, speeding up computations, you should decrease --nsamples instead.
However, it doesn't have that large impact on the quantization time anyway.

from aqlm.

@BlackSamorez Thanks for the answer!

I am trying to quantize finetuned version of mixtral and I had no such long samples (8192) in the training set.

Then should I decrease max_epochs and finetune_max_epochs instead (in order to speedup the process)?

from aqlm.

@VirtualRoyalty you may try and see how shorter sequences affect the quality. When I was tuning Mixtral, i used 7k instead of 8k to fit into memory and this seems to work fine. However, 1k is much shorter than 8k, so I cannot say apriori, whether it matters much.

from aqlm.

@Godofnothing Thanks, good point!

from aqlm.

This issue is stale because it has been open for 30 days with no activity.

from aqlm.

This issue was closed because it has been inactive for 14 days since being marked as stale.

from aqlm.