Why scales need to be transformed by sqrt(scales.max() * scales.min())? about llm-awq HOT 4 CLOSED

Hi, we only care about the relative values of scaling across channels. Multiplying the entire scaling factor vector by a fixed number does not affect the accuracy. However, it helps with numerical stability (a more proper range). Let me know if you have more questions.

from llm-awq.

Hi @MarsJacobs , the most important factor being the accuracy is higher. Our intuition is different compared to SmoothQuant: SmoothQuant wants to preserve the activation outliers for W8A8 quantization; we only want to introduce activation-awareness to the weights. Therefore, we want to use the average to reflect the overall effect from different tokens.

from llm-awq.

Thank you for the further clarification! I have one question. In the AWQ implementation, is there a particular reason for using the abs().mean() of weight and activation to explore the scale value? (For comparison, I understand that SmoothQuant utilizes the max value of weight and activation.)

Additional explanations would be greatly helpful in gaining a deeper understanding of AWQ. Thanks in advance!

from llm-awq.

Thank you for your response. Considering the motivational difference between SmoothQuant's shifting activation outliers and AWQ's activation-aware weight scaling, the reasoning behind the abs().mean() implementation becomes much clearer. Thank you for sharing your great work and for providing further answers!

from llm-awq.