[Question/Feature] Skip initialization after quantization about llm-awq HOT 5 CLOSED

Hi, thanks for the suggestion. We will work on this shortly.

from llm-awq.

Thank you @tonylins and @Sakits. I'm quite certain load times can get down to ~2 seconds, which would be a significant feature since normally it will take 10-20 seconds to load.

from llm-awq.

Hi @casperbh96,

Thank you for your feedback!
Please try the latest version of our AWQ library, it should be a lot faster now.

from llm-awq.

Hi @casperbh96,

Thank you for your feedback! Please try the latest version of our AWQ library, it should be a lot faster now.

Thank you, I will try the latest AWQ.

I also noticed that if I start TinyChat once, we initialize the weights once, then if I CTRL+C and start TinyChat again, it is not re-initializing the weights and goes straight to the chat. Not sure if it's possible to skip initialization entirely by saving initialized weights?

from llm-awq.

Thank you for this update. My model is now initialized in 0.8s. In total, loading is now taking roughly 4 seconds.

from llm-awq.