Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time

The KV cache can exceed the model weights themselves and caps inference batch size; existing compression needed fine-tuning or degraded quality.

The PERSISTENCE OF IMPORTANCE hypothesis: a token that had substantial influence at one decoding step will continue to influence future generations, so influence is predictable from past attention scores. Scissorhands maintains a fixed KV memory budget WITHOUT finetuning by storing PIVOTAL tokens with a higher probability - a Metropolis-style stochastic retention rule rather than a hard top-k - so the budget is respected while pivotal tokens survive.

Paper's own claims (abstract): reduces inference KV-cache memory by UP TO 5x without compromising model quality; and when COMBINED WITH 4-bit quantization (which is for weights, not the cache) achieves up to 20x compression. The 5x is the standalone KV claim; the 20x only exists in combination with weight quantization, so quoting 20x as the KV-cache result would misrepresent the paper.

arXiv ID 2305.17118 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. CORRECTION TO THE SOURCE RECORD: it says 'arXiv preprint; no journal_ref', but OpenReview returns venue='NeurIPS 2023 poster', venueid='NeurIPS.cc/2023/Conference' for the exact title. venue=neurips, year=2023. Watch for a title-collision trap: the same OpenReview search surfaces a DIFFERENT NeurIPS paper, 'Scissorhands: Scrub Data Influence via Connection Sensitivity' (data influence, DBLP CoRR 2024) - unrelated to KV-cache compression despite the identical name prefix. adoption=research-only verified by absence: no Scissorhands class or policy in transformers cache_utils.py, vLLM's v1 cache layer, or SGLang. The stochastic retention rule needs to run inside the decode attention path, and no mainstream engine exposes a per-token retention hook. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.