inference/research
verifiedmedium · 0.80updated 2026-10-04

H2O (heavy-hitter KV eviction) - NOT KV quantization

none - no bits are reduced. Evicts tokens instead. Retains a configurable budget of heavy-hitter plus recent tokens (20% heavy hitters in the paper's throughput results).

Fields

id
quantization/h2o-kv-eviction
Scheme
kv-eviction
Bits
none - no bits are reduced. Evicts tokens instead. Retains a configurable budget of heavy-hitter plus recent tokens (20% heavy hitters in the paper's throughput results).
Activation scheme
None. The mechanism is KV cache EVICTION: a dynamic submodular formulation retains a balance of recent and high-attention-score 'heavy hitter' tokens. Nothing is quantized; entries are dropped entirely.
Kernels
Runs on top of existing inference engines (the paper evaluates against DeepSpeed Zero-Inference, HuggingFace Accelerate and FlexGen).
Quality delta
From the abstract: with 20% heavy hitters, H2O improves throughput over DeepSpeed Zero-Inference by up to 29x, HuggingFace Accelerate by up to 29x, and FlexGen by up to 3x on OPT-6.7B and OPT-30B; at the same batch size it reduces latency by up to 1.9x. Validated across OPT, LLaMA and GPT-NeoX. The 29x figures are against systems whose baselines are far weaker, so they mostly measure the baseline gap, not H2O.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Recorded explicitly to PREVENT A CONFUSION, which is why it exists. H2O is the canonical example of a KV-cache method that is not quantization: it reduces KV memory by discarding tokens, not by reducing bits. Every bit saved is a token of context thrown away permanently, which is a qualitatively different trade from quantization's lossy-but-uniform compression. This matters for capacity planning: quantized KV degrades gracefully, evicted KV cannot be recovered. The paper also establishes the heavy-hitter observation that later KV-quantization and KV-sparsity work all build on. scheme is a deliberate placeholder - request the parent add 'kv-eviction' and 'kv-sparsity' enum values. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Recorded explicitly to PREVENT A CONFUSION, which is why it exists. H2O is the canonical example of a KV-cache method that is not quantization: it reduces KV memory by discarding tokens, not by reducing bits. Every bit saved is a token of context thrown away permanently, which is a qualitatively different trade from quantization's lossy-but-uniform compression. This matters for capacity planning: quantized KV degrades gracefully, evicted KV cannot be recovered. The paper also establishes the heavy-hitter observation that later KV-quantization and KV-sparsity work all build on. scheme is a deliberate placeholder - request the parent add 'kv-eviction' and 'kv-sparsity' enum values. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)