SnapKV: LLM Knows What You are Looking for Before Generation

The KV cache grows with input length, inflating both memory and per-token decode cost, and existing compression methods need fine-tuning or hurt quality on real prompts.

Fine-tuning-free prompt compression exploiting a regularity: each attention head consistently focuses on the same prompt attention features during generation, and that pattern is already visible in a small OBSERVATION WINDOW at the end of the prompt. SnapKV pools keys/values in that window, uses attention from the observation window to score all earlier positions, then keeps clustered high-importance KV positions per head plus the observation window itself, as a prompt-only prefix compression applied once before generation.

Paper's own claims (abstract): a 3.6x increase in GENERATION SPEED and 8.2x improvement in MEMORY EFFICIENCY vs the baseline when processing 16K-token inputs; and 380K context tokens processed on a single A100-80GB with the HuggingFace implementation, with a negligible accuracy drop on needle-in-a-haystack. Comparable performance to baselines across 16 long-sequence datasets.

arXiv ID 2404.14469 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. CORRECTION TO THE SOURCE RECORD: it says 'arXiv preprint; no journal_ref' with no venue evidence, but OpenReview returns venue='NeurIPS 2024 poster', venueid='NeurIPS.cc/2024/Conference' for the exact title. venue=neurips, year=2024. DEDUP CHECK: the source note says 'A separate verified quantization record also exists under the bare id snapkv (owned by another slice)'. I checked - there is NO data/papers/snapkv.json. What exists is data/quantization/snapkv.json, which is a DIFFERENT record type (a quantization scheme). No duplicate paper record exists, so nothing to delete; the -paper suffix on this slug is the correct disambiguation and should stay. adoption=research-only verified by absence: no SnapKV class among the 24 cache classes in transformers cache_utils.py, no snapkv reference in vLLM's v1 cache layer files, and no SGLang equivalent. Note SnapKV's mechanism is PREFIX compression applied once before decode, which is structurally different from H2O/Scissorhands eviction DURING decode - it survives as a benchmark (LongBench) method rather than a serving path, and that difference is why 'research-only' is right even though the paper is well known. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.