AWQ INT4-g128 WikiText-2 perplexity delta vs fp16, Mistral-7B
0.16 perplexity_delta (WikiText-2 ppl points, quantized minus FP16)
Fields
id
benchmarks/awq-int4g128-wikitext2-ppl-delta-mistral-7bMetric
quality
Value
0.16
Unit
perplexity_delta (WikiText-2 ppl points, quantized minus FP16)
Model
Mistral-7B-Instruct-v0.2
Methodology
AWQ paper Table 5. Model Mistral-7B-Instruct-v0.2, weight-only grouped INT4, group size 128, activations FP16. Metric: WikiText-2 perplexity as measured by the AWQ authors. FP16 baseline 4.14 in the same table; AWQ INT4-g128 scores 4.30. The same table gives INT3-g128 at 4.83 (+0.69).
Measured by
third_party
Reproducible
yes
notes
DELTA = +0.16 ppl (4.30 vs 4.14). This record carries the format-family generalisation claim: the same AWQ recipe, same group size, applied to a GQA model newer than anything in the paper's main tables. What the same table shows for Mixtral-8x7B-Instruct-v0.1 at INT4-g128 is 5.94 -> 6.05 (+0.11), while at INT3-g128 it jumps to 6.52 (+0.58). MoE models degrade roughly 4x faster than a dense model of comparable size when dropping from 4 to 3 bits, so 'INT3 is fine' does not port across architectures. Caution: the paper does not state its eval harness version, so these absolute perplexities are not comparable to the Llama-3.1-8B numbers from the GGUF k-quant study, which used a different model and a different perplexity implementation.
Notes
DELTA = +0.16 ppl (4.30 vs 4.14). This record carries the format-family generalisation claim: the same AWQ recipe, same group size, applied to a GQA model newer than anything in the paper's main tables. What the same table shows for Mixtral-8x7B-Instruct-v0.1 at INT4-g128 is 5.94 -> 6.05 (+0.11), while at INT3-g128 it jumps to 6.52 (+0.58). MoE models degrade roughly 4x faster than a dense model of comparable size when dropping from 4 to 3 bits, so 'INT3 is fine' does not port across architectures. Caution: the paper does not state its eval harness version, so these absolute perplexities are not comparable to the Llama-3.1-8B numbers from the GGUF k-quant study, which used a different model and a different perplexity implementation.