inference/research
verifiedmedium · 0.78updated 2026-10-04

Pensieve (multi-tier conversation-state KV cache) - NOT KV quantization

none - no bits reduced. Retains and reuses conversation state across requests in a multi-tier cache.

Fields

id
quantization/pensieve
Scheme
kv-eviction
Bits
none - no bits reduced. Retains and reuses conversation state across requests in a multi-tier cache.
Activation scheme
None. Maintains conversation state across requests so multi-turn history is not reprocessed; a multi-tier caching strategy uses both GPU and CPU memory, and the paper generalizes PagedAttention to support attention against a GPU cache spread over non-contiguous memory.
Kernels
Generalizes the PagedAttention kernel to attention over multiple input tokens against non-contiguous GPU cache.
Quality delta
From the abstract: 1.14x-3.0x the throughput of vLLM and TensorRT-LLM with significantly reduced latency, in the multi-turn conversation setting.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Boundary record, and the clearest statement of the non-quantization case: Pensieve solves the SAME problem as KIVI - a multi-turn conversation's KV cache does not fit - by storing it across tiers instead of compressing it. Since lossless storage is strictly better than lossy compression when the constraint is host memory rather than GPU memory, Pensieve is the correct answer when an offload path exists and KIVI is the correct answer when it does not. Including it so a reader does not conclude that the whole multi-turn problem requires quantization. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: eviction reduces HOW MANY KV positions are retained; no bits per retained value change, so there is no weight grid to group. The null here is a fact about the method, not missing data. Boundary record, and the clearest statement of the non-quantization case: Pensieve solves the SAME problem as KIVI - a multi-turn conversation's KV cache does not fit - by storing it across tiers instead of compressing it. Since lossless storage is strictly better than lossy compression when the constraint is host memory rather than GPU memory, Pensieve is the correct answer when an offload path exists and KIVI is the correct answer when it does not. Including it so a reader does not conclude that the whole multi-turn problem requires quantization. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-eviction'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)