inference/research
verifiedmedium · 0.80updated 2026-10-04

PyramidKV (layer-dependent KV budget) - NOT KV quantization

none - allocates different KV cache SIZES per layer (more in lower layers, less in higher ones) rather than reducing bits.

Fields

id
quantization/pyramidkv
Scheme
kv-eviction
Bits
none - allocates different KV cache SIZES per layer (more in lower layers, less in higher ones) rather than reducing bits.
Activation scheme
None. Based on Pyramidal Information Funneling: attention scatters widely in lower layers, consolidates in middle layers, and concentrates on a few critical tokens in higher layers. The KV budget is therefore allocated inversely - more cache low, less cache high - diverging from uniform-budget methods.
Kernels
Evaluated on LongBench and Needle-in-a-Haystack; no specific kernel named in the abstract.
Quality delta
From the abstract: matches full-KV-cache performance on LongBench while retaining only 12% of the KV cache; at an extreme 0.7% retention it surpasses other KV compression techniques with up to 20.5 absolute accuracy improvement on TREC; retaining just 128 KV entries gives LLaMA-3-70B 100.0 accuracy on Needle-in-a-Haystack.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Boundary record, and the most surprising one: 128 total KV entries is a compression ratio so extreme that it stops being about compression and becomes about caching only attention sinks. That is a real finding about where attention information actually lives, and it is worth reading alongside KIVI - if a handful of entries carry most of the attention mass, then spending 4 bits per entry on the rest is largely wasted. Boundary stated plainly because the 12%-of-KV-cache phrasing sounds like a bit-width claim and is not one. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Boundary record, and the most surprising one: 128 total KV entries is a compression ratio so extreme that it stops being about compression and becomes about caching only attention sinks. That is a real finding about where attention information actually lives, and it is worth reading alongside KIVI - if a handful of entries carry most of the attention mass, then spending 4 bits per entry on the rest is largely wasted. Boundary stated plainly because the 12%-of-KV-cache phrasing sounds like a bit-width claim and is not one. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)