4bit kernel error when large input size about llm-awq HOT 4 OPEN

Hi @Tracin! Thanks a lot for bringing the issue to our attention. We have fixed the problem in #8 , please have a look! Once again, thank you for your interest in AWQ.

from llm-awq.

Hi @ys-2020! Very appreciated for your MR for this. It is very impressive both the AWQ method and this fast kernel.
However I am afraid there is still something wrong in it. When I continue doubling the M up to 2048 * 32, I met a CUDA error: an illegal memory access was encountered.
I believe you understand it is 32 batchsize and 2k sentence length which is quite often used for LLM inference.

from llm-awq.

Hi @Tracin , we have reproduced the illegal memory access issue and are trying to fix it. Please stay tuned for our updates. Thanks!

from llm-awq.

I got this error today as well for long generation. Do you know what the issue is, and if there is a patch / fix we can do locally?

Thanks!

from llm-awq.