AWQ INT4-g128 WikiText-2 perplexity delta vs fp16, Llama-2 7B
0.13 perplexity_delta (WikiText-2 ppl points, quantized minus FP16)
Fields
id
benchmarks/awq-int4g128-wikitext2-ppl-delta-llama2-7bMetric
quality
Value
0.13
Unit
perplexity_delta (WikiText-2 ppl points, quantized minus FP16)
Model
Llama-2 7B
Methodology
AWQ paper Table 4. Model Llama-2 7B, weight-only grouped INT4, group size 128 (the paper's default throughout), activations left in FP16. Calibration: small set from The Pile, grid size 20 for the alpha search. Metric: WikiText-2 test perplexity, computed with the paper's own harness on the quantized checkpoint. Eval settings: no few-shot prompting - pure language-modeling perplexity, not a downstream benchmark. FP16 baseline for the same model in the same table is 5.47; AWQ INT4-g128 scores 5.60.
Measured by
third_party
Reproducible
yes
notes
DELTA = +0.13 ppl (5.60 quantized vs 5.47 fp16), derived from two cells both printed in Table 4. How easy this is to misread: the same row also lists RTN at 5.73 (+0.26), GPTQ at 5.69 (+0.22) and GPTQ-R at 5.63 (+0.16), so '4-bit costs you X' has four defensible answers depending on which quantizer you mean. The paper's selling point is that AWQ beats the other three, but every one of them is within a quarter of a perplexity point of fp16 - the margin that distinguishes the methods is smaller than the margin that distinguishes any of them from full precision. Note the direction: this is a WEIGHT-ONLY 4-bit record. AWQ quantizes no activations, so it says nothing about 4-bit activations.
Notes
DELTA = +0.13 ppl (5.60 quantized vs 5.47 fp16), derived from two cells both printed in Table 4. How easy this is to misread: the same row also lists RTN at 5.73 (+0.26), GPTQ at 5.69 (+0.22) and GPTQ-R at 5.63 (+0.16), so '4-bit costs you X' has four defensible answers depending on which quantizer you mean. The paper's selling point is that AWQ beats the other three, but every one of them is within a quarter of a perplexity point of fp16 - the margin that distinguishes the methods is smaller than the margin that distinguishes any of them from full precision. Note the direction: this is a WEIGHT-ONLY 4-bit record. AWQ quantizes no activations, so it says nothing about 4-bit activations.