KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

Quantified low-bit KV cache. Its abstract names the exact two-part failure this repo records: with larger batch sizes and longer contexts the KV cache 'significantly increases memory demands and becomes the new bottleneck in speed and memory usage', AND 'the loading of the KV cache causes the computational core to be idle, which limits the inference speed' - the second clause is the arithmetic-intensity statement that decode attention stays ~1 flop/byte regardless of context or batch. Asymmetric per-channel key / per-token value at 2 bits is the extreme version of the fp8/int8 KV fix named on the decode-attention record. | SOURCE AUDIT 2026-10-03: the 'publisher' field held a paragraph of prose (authors + affiliations, 87 chars) instead of a publisher name; trimmed to 'arXiv' with the authorship detail moved here.