How to measure the speedup of W4A16 kernel like Figure 6？ about llm-awq HOT 4 OPEN

+1

from llm-awq.

+1

from llm-awq.

+1

from llm-awq.

Hi all, thank you for your interests in our work!

In our speed benchmark, we based on the generation stage starting from zero context and measured the median latency of generating 2048 tokens, since evaluating speed using the WikiText2 perplexity evaluation may not accurately reflect the practical usage of LLM.

And it's necessary to replace some inefficient FP16 kernels (such as layernorm and rotary position embedding) in the HuggingFace implementation when testing the speed. At present, we have not released a version for speed testing yet. Please stay tuned for our further updates.

from llm-awq.