inference/research
verifiedmedium · 0.80updated 2026-10-04

SnapKV (observation-window KV selection) - NOT KV quantization

none - selects important KV POSITIONS per attention head from an observation window; no bits are reduced.

Fields

id
quantization/snapkv
Scheme
kv-eviction
Bits
none - selects important KV POSITIONS per attention head from an observation window; no bits are reduced.
Activation scheme
None. Fine-tuning-free: each attention head is observed to focus on stable prompt-attention features during generation, and this pattern can be read off an 'observation window' at the end of the prompt. SnapKV then selects clustered important KV positions per head.
Kernels
Runs in HuggingFace implementations with minor changes, and fits 380K context tokens on a single A100-80GB.
Quality delta
From the abstract: 3.6x increase in generation speed and 8.2x memory-efficiency improvement over baseline at 16K-token inputs, comparable performance across 16 long-sequence datasets, and only a negligible accuracy drop on Needle-in-a-Haystack.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Boundary record. SnapKV's fine-tuning-free property is what distinguishes it operationally from learned approaches - it works on any checkpoint with no retraining and no calibration pipeline, which is why it spread into serving stacks faster than most of this literature. Its observation-window trick is the same underlying fact H2O identified (attention concentrates) applied at the prompt level rather than the token level. The 8.2x memory-efficiency figure at 16K tokens is a prompt-length-specific measurement; memory savings from eviction scale with context length, so quoting it outside that regime would be misleading. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Boundary record. SnapKV's fine-tuning-free property is what distinguishes it operationally from learned approaches - it works on any checkpoint with no retraining and no calibration pipeline, which is why it spread into serving stacks faster than most of this literature. Its observation-window trick is the same underlying fact H2O identified (attention concentrates) applied at the prompt level rather than the token level. The 8.2x memory-efficiency figure at 16K tokens is a prompt-length-specific measurement; memory savings from eviction scale with context length, so quoting it outside that regime would be misleading. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)