AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Weight-only quantization to 3-4 bits either degrades quality badly or requires expensive per-channel/group grid search at quantization time; naive round-to-nearest INT4 is much worse than RTN at INT8.

Protect a small subset (~1%) of weight channels that are salient in the ACTIVATION distribution rather than the weight distribution: score each input channel by the mean magnitude of its activations across calibration data, scale the salient channels up by a per-channel factor s, quantize with per-group asymmetric INT3/INT4 grids, and fold s into the preceding LayerNorm so no extra runtime cost. Grid scales chosen by a search that minimises the output reconstruction error weighted by activation magnitude.

Paper's own claim (abstract + tables 1-5, MLsys 2024 Best Paper): near-lossless W4 (group size 128) on LLaMA/LLaMA-2 families with 3.6x smaller weights, plus 4-bit models fitting on a single 24GB-class GPU; measured decode speedup 1.7-3.9x over FP16 on the same GPU. COMPRESSION RATIO: ~4x weight compression (16 -> 4 bit, g128). QUALITY COST: INT4-g128 WikiText perplexity is within noise of FP16 for most LLaMA models; INT3-g128 is NOT - the perplexity gap widens sharply there, which the paper reports rather than hides.

SLUG RENAMED 2026-10-04 (parent agent): was 'awq', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs (['concepts', 'format_id', 'scheme', 'supported_formats']) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2306.00978 verified via /abs/. Venue: MLSys 2024, from the arXiv comment field ('MLSys 2024 Best Paper Award') plus Semantic Scholar. Year = venue year 2024 (v1 was 2023-06-01). The '1.7-3.9x' speedup is the paper's own decode measurement and is batch-size dependent - it is not a prefill number. arXiv ID and title verified by fetching https://arxiv.org/abs/2306.00978 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2306.00978) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: mlsys has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.