KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

KV cache activations become the dominant memory consumer at long context, and existing solutions fail to represent them accurately below 4 bits.

Four components: (1) PER-CHANNEL key quantization - change the dimension along which keys are quantized to match their distribution; (2) PRE-RoPE key quantization - quantize keys BEFORE the rotary positional embedding, so RoPE's per-position rotation cannot smear values across quantization buckets; (3) non-uniform, per-layer sensitivity-weighted datatypes so each layer gets the codebook that suits it; (4) per-vector dense-and-sparse decomposition isolating outliers for each vector rather than globally.

Paper's own claim (abstract): on LLaMA / Llama-2 / Llama-3 / Mistral, LESS THAN 0.1 PERPLEXITY DEGRADATION with 3-bit quantization on both WikiText-2 and C4, outperforming existing approaches. COMPRESSION RATIO: 3-bit KV cache vs FP16 KV cache = ~5.3x cache reduction. Consequence claim: enables serving LLaMA-7B at up to 1 MILLION context on a SINGLE A100-80GB, and up to 10 million context across 8 A100-80GBs. QUALITY COST: <0.1 perplexity - the <0.1 figure is the headline and it is unusually strong.

SLUG RENAMED 2026-10-04 (parent agent): was 'kvquant', colliding with the quantization record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs ([]) all mean the quantization record, so none needed repointing; papers are cited via `sources`, never by bare slug. arXiv ID 2401.18079 verified via /abs/; v6. Venue NeurIPS 2024 confirmed by the arXiv comment field and Semantic Scholar; year = venue year 2024. hardware_relevance names the A100-80GB because the 1M-context single-GPU claim is explicitly an A100-80GB claim. Repo already has source qlab-kvquant-paper. arXiv ID and title verified by fetching https://arxiv.org/abs/2401.18079 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2401.18079) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.