inference/research
verifiedmedium · 0.80updated 2026-10-04

ZipCache (channel-separable tokenwise KV quantization + salient token identification)

mixed precision across salient and non-salient tokens; headline is a 4.98x KV compression ratio, not a single bit width

Fields

id
quantization/zipcache
Scheme
kv-cache-quant
Bits
mixed precision across salient and non-salient tokens; headline is a 4.98x KV compression ratio, not a single bit width
Activation scheme
Not an activation scheme - KV cache. Two mechanisms: (1) CHANNEL-SEPARABLE TOKENWISE quantization - quantizing per token but keeping scales separate per channel, which substantially reduces quantization-parameter memory overhead versus fine-grained groupwise schemes (this matters because at 2-3 bits the scales themselves become a significant fraction of the cache); (2) normalized attention score as a saliency metric exploiting the lower-triangle structure of the attention matrix, with an efficient approximation that decouples the metric from full attention scores so it stays compatible with FlashAttention-style kernels.
Kernels
Designed explicitly for compatibility with fast attention implementations like FlashAttention - a constraint most KV-quantization papers ignore and which determines whether the method survives contact with a real serving stack.
Quality delta
From the abstract: on Mistral-7B / GSM8k, compresses the KV cache by 4.98x with only a 0.38% drop in accuracy. On LLaMA3-8B at input length 4096: 37.3% reduction in prefill-phase latency, 56.9% reduction in decoding-phase latency, 19.8% reduction in GPU memory usage.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Third KV record; same enum mismatch. Note the name collision risk: there is an unrelated DRAM/SSD key-value-cache systems paper also called ZipCache (arXiv 2411.03174, 'A DRAM/SSD Cache with Built-in Transparent Compression'). Different field entirely - that one is about storage hardware, not quantization. This record is the KV quantization paper (2405.14256); the other is explicitly out of scope for this repo's quantization type. The idea worth stealing is that scale metadata is not free: at 2-3 bits the scales can dominate, so choosing the grouping axis to minimize scale overhead (channel-separable rather than fine-grained groupwise) is a first-order decision, not a detail. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: this scheme quantizes the KV cache, not the weights, so there are no weight groups to size. A number in this field would be a category error. (Where the underlying method has a real grouping - e.g. KIVI per-channel keys, KVQuant per-channel - that lives in the method's own record, not in weight_group_size.) Third KV record; same enum mismatch. Note the name collision risk: there is an unrelated DRAM/SSD key-value-cache systems paper also called ZipCache (arXiv 2411.03174, 'A DRAM/SSD Cache with Built-in Transparent Compression'). Different field entirely - that one is about storage hardware, not quantization. This record is the KV quantization paper (2405.14256); the other is explicitly out of scope for this repo's quantization type. The idea worth stealing is that scale metadata is not free: at 2-3 bits the scales can dominate, so choosing the grouping axis to minimize scale overhead (channel-separable rather than fine-grained groupwise) is a first-order decision, not a detail. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'bitsandbytes-nf4' (a proxy recorded while the enum lacked a legal value) and is now 'kv-cache-quant'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)