inference/research
verifiedmedium · 0.78updated 2026-10-04

ShadowKV (low-rank key cache on GPU, value cache offloaded)

none by quantization; keys are compressed by a LOW-RANK representation and values are offloaded to CPU rather than reduced in precision.

Fields

id
quantization/shadowkv
Scheme
kv-cache-quant
Bits
none by quantization; keys are compressed by a LOW-RANK representation and values are offloaded to CPU rather than reduced in precision.
Activation scheme
None. Stores the low-rank key cache on GPU and offloads the value cache, with an accurate KV selection strategy that reconstructs minimal sparse KV pairs on the fly to keep decoding latency low.
Kernels
bytedance/ShadowKV implementation, evaluated on A100.
Quality delta
From the abstract: supports up to 6x larger batch sizes and boosts throughput by up to 3.04x on an A100 GPU without sacrificing accuracy, evaluated on RULER, LongBench and Needle-In-A-Haystack across Llama-3.1-8B, Llama-3-8B-1M, GLM-4-9B-1M, Yi-9B-200K, Phi-3-Mini-128K and Qwen2-7B-128K.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Boundary record, and the one with the most interesting overlap with the rotation family: ShadowKV's low-rank key representation and QuaRot/SpinQuant's rotation-based approaches are both attacks on the fact that the key cache is highly redundant. Low rank says the redundancy is linear; incoherent rotation says it is an outlier problem. The paper's motivation is also a critique of prior sparse-attention work - that those methods either failed to cut GPU memory enough or paid significant decoding latency from CPU offloading - which is why the on-the-fly sparse reconstruction matters. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'spinquant' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Boundary record, and the one with the most interesting overlap with the rotation family: ShadowKV's low-rank key representation and QuaRot/SpinQuant's rotation-based approaches are both attacks on the fact that the key cache is highly redundant. Low rank says the redundancy is linear; incoherent rotation says it is an outlier problem. The paper's motivation is also a critique of prior sparse-attention work - that those methods either failed to cut GPU memory enough or paid significant decoding latency from CPU offloading - which is why the on-the-fly sparse reconstruction matters. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'spinquant' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)