QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Activation outliers in LLMs make W4A4 (and even KV4) quantization collapse, and every fix so far keeps a handful of channels in higher precision.

Computational invariance: inserting an orthogonal matrix Q into the residual stream is a no-op at full precision because Q^T Q = I, and any residual is orthogonal to itself, so the model output is unchanged while the outlier structure is redistributed across channels. QuaRot applies randomized Hadamard rotations end-to-end - to the embedding output, to the residual write/read paths, to the Q/K head inputs, and to the KV cache - so that all matmuls (weights, activations AND KV cache) can run in 4 bits with NO channel retained in higher precision.

Paper's own claim (abstract, LLaMA2-70B, 4-bit weights + activations + KV cache): at most 0.47 WikiText-2 perplexity loss and 99% of zero-shot performance retained. COMPRESSION RATIO: 4 bits on all three tensors, i.e. ~4x weight and ~4x KV-cache reduction vs FP16 (8x if counting weights+activations together). Also claims LOSSLESS 6-bit and 8-bit LLaMA2 with round-to-nearest and NO calibration data at all. This 0.47-ppl figure is the single most-cited number in the 4-bit LLM quantization literature and it is the paper's own, on one model family.

SLUG RENAMED 2026-10-04 (parent agent): was 'quarot', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs (['scheme']) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2404.00456 verified via /abs/; v2. Venue NeurIPS 2024 confirmed by Semantic Scholar. Year = venue year 2024. adoption=in-upstream-engine rather than in-production: the Hadamard-rotation primitives are in vLLM and TorchAO, but a stock W4A4KV4 QuaRot checkpoint is not the default path anyone serves. The 99%-of-zero-shot claim is a ratio, not a pointwise accuracy delta - do not read it as 'no quality loss'. arXiv ID and title verified by fetching https://arxiv.org/abs/2404.00456 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2404.00456) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.