A question about the metrics in the paper about llm-awq HOT 6 CLOSED

here's the performance:

AWQ is more hardware efficient and simpler to implement than SpQR, but the compression ratio seems to be worse than SpQR.

from llm-awq.

Hi @loklok-infi,

Thanks for your interests in our work!
I think there are two potential reasons for the difference in the results:
(i) We used lm-eval-harness for evaluation, while GPTQ used their own implementation for evaluation (please refer here). There might be some differences in the experiment settings between them.
(ii) Regarding the 3/4-bit results, the results from GPTQ's paper are based on per-channel quantization without using group quantization. Our results are based on group quantization with a group size of 128.
Hope this answers your question :)

from llm-awq.

Hi @loklok-infi,

Thanks for your interests in our work! I think there are two potential reasons for the difference in the results: (i) We used lm-eval-harness for evaluation, while GPTQ used their own implementation for evaluation (please refer here). There might be some differences in the experiment settings between them. (ii) Regarding the 3/4-bit results, the results from GPTQ's paper are based on per-channel quantization without using group quantization. Our results are based on group quantization with a group size of 128. Hope this answers your question :)

Thank you for answering! Actually what confuses me more is the fp16 results are also ~10% different, but as you said I guess it's from the lm-eval-harness and the implementation of GPTQ.

I guess it's a problem for the whole community today, a similar problem seems happened between huggingface's LLM leaderboard and LLaMA's official result, hope soon we could have a one-true-standard benchmark implementation for LLMs.

from llm-awq.

here's the performance:

LLaMA-7B BPW Wikitext2
FP16 16 5.68
RTN 4.00 6.29
RTN 3.00 25.54
GPTQ-128g 4.15 5.85
GPTQ-128g 3.15 6.61
AWQ-128g 4.15 5.81
AWQ-128g 3.15 6.46
AWQ-32g 3.60 6.10
SpQR-3b-16g-3b-32g-0.4% 3.63 5.73
AWQ is more hardware efficient and simpler to implement than SpQR, but the compression ratio seems to be worse than SpQR.

Thank you! It's very helpful👍

from llm-awq.

here's the performance:

LLaMA-7B BPW Wikitext2
FP16 16 5.68
RTN 4.00 6.29
RTN 3.00 25.54
GPTQ-128g 4.15 5.85
GPTQ-128g 3.15 6.61
AWQ-128g 4.15 5.81
AWQ-128g 3.15 6.46
AWQ-32g 3.60 6.10
SpQR-3b-16g-3b-32g-0.4% 3.63 5.73
AWQ is more hardware efficient and simpler to implement than SpQR, but the compression ratio seems to be worse than SpQR.

Hey do you have a repo to benchmark all these quant methods?

from llm-awq.

here's the performance:
LLaMA-7B BPW Wikitext2
FP16 16 5.68
RTN 4.00 6.29
RTN 3.00 25.54
GPTQ-128g 4.15 5.85
GPTQ-128g 3.15 6.61
AWQ-128g 4.15 5.81
AWQ-128g 3.15 6.46
AWQ-32g 3.60 6.10
SpQR-3b-16g-3b-32g-0.4% 3.63 5.73
AWQ is more hardware efficient and simpler to implement than SpQR, but the compression ratio seems to be worse than SpQR.

Hey do you have a repo to benchmark all these quant methods?

Yes , Code for this benchmark will be appreciated

from llm-awq.