KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

KV cache quantization is applied per-token to both keys and values, which is exactly wrong for keys and wastes most of the value cache's headroom.

A distribution study of KV elements on LLaMA / Falcon / Mistral establishes the asymmetry that drives everything: KEYS must be quantized PER-CHANNEL (group elements along the channel dimension) while VALUES must be quantized PER-TOKEN. Implementation matches: per-channel key quantization and per-token value quantization, each with its own group size, tuned-free. The channel/token split also happens to align with how the attention kernel reads memory, so it is hardware-coherent rather than merely statistically better.

Paper's own claim (abstract): LLaMA / Falcon / Mistral 'maintain almost the same quality while using 2.6x LESS PEAK MEMORY (including model weight)'. COMPRESSION RATIO: 2-bit KV cache vs FP16 = 8x on the cache itself; the 2.6x figure is whole-model peak memory INCLUDING weights, which is the more honest number and the one to quote. QUALITY COST: 'almost the same quality' - not a paired delta in the abstract.

arXiv ID 2402.02750 verified via /abs/; v2. Venue ICML 2024 confirmed by the arXiv comment field ('ICML2024') and Semantic Scholar; year = venue year 2024. IMPORTANT reading note: the headline 2.6x is whole-model peak memory including weights - quoting it as a '2.6x KV compression' would be wrong, the cache itself is ~8x. arXiv ID and title verified by fetching https://arxiv.org/abs/2402.02750 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2402.02750) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.