Merge the more models branch with main branch about llm-awq HOT 4 CLOSED

Hey @Sakits,

Can you please look at the open PR #36 and merge it if everything looks good? I have extensively tested it and have even released multiple quantized models on HuggingFace (link).

from llm-awq.

Hi @abhinavkulkarni,

The quantized models you've shared on HuggingFace are impressive! Thank you for your valuable contribution!
My bandwidth has been limited in recent times. I will prioritize reviewing and merging your PR as soon as I can.

from llm-awq.

Hey @Sakits,

Is there any plan to merge the dev/more_models branch with main?

from llm-awq.

Hey @Sakits,

Is there any plan to merge the dev/more_models branch with main?

#41 it is already merged, main works with the new models that were implemented. Try out tinychat, it's roughly 2x the speed of using AWQ with accelerate - it still has more room for optimization though, only LLaMa-2 models are optimized to run at 100+ tokens/s for 7B models consistently.

from llm-awq.