aqlm/inference_kernels/cuda_kernel.cu compilation errors about aqlm HOT 10 CLOSED

Please provide the following information:

• aqlm version
• torch version
• CUDA version
• Your GPU model

from aqlm.

Thanks!

aqlm==1.1.3
torch: 2.2.2+cu121
cuda: 12.1
GPU: Tesla V100 via Databricks: NCv3 VM https://learn.microsoft.com/en-us/azure/virtual-machines/ncv3-series

(And FYI using ISTA-DASLab/Mixtral-8x7B-Instruct-v0_1-AQLM-2Bit-1x16-hf, I'm still having the same issue)

from aqlm.

@amrothemich v100 doesn't really support efficient bf16 operations. If you were to update aqlm to the latest version it would perform a check for Compute Capability to display a more readable error.

from aqlm.

Okay got it, thanks. So is bf16 a requirement for the whole library or just this model?

from aqlm.

bfloat16 is not a requirement at all. You can pass torch_dtype=torch.float16 to from_pretrained to use standard half precision. It should work for any model no problem.

from aqlm.

Actually, float16 is the default for the models we put out. No need to specify it explicitly.
But first please update aqlm. It simply wouldn't compile otherwise.

from aqlm.

from aqlm.

I see, I'm sorry. It's not that you didn't update, it's actually the opposite: I broke it in the latest release.
The latest dequantization kernels won't compile on GPUs with Compute Capability of 8 or less.
For now, you can downgrade to 1.1.2 and everything should work. I'll try and fix the error to allow you to use the latest dequantization kernels as well.

from aqlm.

from aqlm.

Please update to aqlm>=1.1.4 and it should resolve the issue.
Feel free to reopen it if it doesn't.

from aqlm.