33B llama quantization post-inference time about aqlm HOT 3 CLOSED

Hi @ACoder-J !
The inference speed is really bottlenecked by transformers for multiple reasons (transformers being slow and kernel launches taking long).
One way to achieve proper speedup is to compile your model into a cuda graph. It's a new feature added to PyTorch in 2.0.0 and refined in 2.2.1. People of transformers have also been working hard to properly support and document it. To check out what you can do with it already, please refer to this AQLM demo.
Another way to properly serve AQLM is to use the vllm integration. It's not complete yet but it might be usable already (not sure).

from aqlm.

This issue is stale because it has been open for 30 days with no activity.

from aqlm.

This issue was closed because it has been inactive for 14 days since being marked as stale.

from aqlm.