inference/research
verifiedmedium · 0.80updated 2026-10-04

DuoAttention (head-wise retrieval/streaming KV sparsity) - NOT KV quantization

none by default - retrieval heads keep a full KV cache, streaming heads use a lightweight CONSTANT-LENGTH KV cache. The paper's headline long-context result is achieved combined WITH quantization.

Fields

id
quantization/duoattention-kv-sparsity
Scheme
kv-sparsity
Bits
none by default - retrieval heads keep a full KV cache, streaming heads use a lightweight CONSTANT-LENGTH KV cache. The paper's headline long-context result is achieved combined WITH quantization.
Activation scheme
None. Heads are classified into retrieval heads (need full attention over all tokens) and streaming heads (attend to recent tokens and attention sinks only). Only retrieval heads get the full cache; streaming heads get a fixed small cache regardless of context length.
Kernels
mit-han-lab/duo-attention implementation.
Quality delta
From the abstract: reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA, speeds decoding by up to 2.18x (MHA) and 1.50x (GQA), and accelerates pre-filling by up to 1.73x (MHA) and 1.63x (GQA), with minimal accuracy loss. Combined with quantization, it enables LLaMA-3-8B decoding at 3.3 million context length on a single A100 GPU.
notes
GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: page/head-level selection of which KV to keep or prefetch. No weights are re-gridded. Third boundary record, and the most useful one for planning because it demonstrates the two mechanisms COMPOSE. The 3.3M-token-on-one-A100 figure is explicitly 'combined with quantization' - sparsity alone does not get there. Practical reading: per-head structure means a uniform KV bit width is leaving value on the table, since most heads do not need full history at all. The classification is cheap (optimization-based, synthetic data, no retraining), which is why it is deployable on models whose training data you do not have. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'spinquant' (a proxy recorded while the enum lacked a legal value) and is now 'kv-sparsity'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Notes

GROUP SIZE NOT APPLICABLE BY CONSTRUCTION: page/head-level selection of which KV to keep or prefetch. No weights are re-gridded. Third boundary record, and the most useful one for planning because it demonstrates the two mechanisms COMPOSE. The 3.3M-token-on-one-A100 figure is explicitly 'combined with quantization' - sparsity alone does not get there. Practical reading: per-head structure means a uniform KV bit width is leaving value on the table, since most heads do not need full history at all. The classification is cheap (optimization-based, synthetic data, no retraining), which is why it is deployable on models whose training data you do not have. SCHEME RECLASSIFIED (parent agent, 2026-10-03): scheme was 'spinquant' (a proxy recorded while the enum lacked a legal value) and is now 'kv-sparsity'. NATIVE_SUPPORT CLASSIFIED 2026-10-04: this field is still [] and that is CORRECT, but the empty list used to be ambiguous between three different states and now is not. Reason class on this record: KV-CACHE-ONLY, SO STRUCTURALLY ABSENT. The bits field records that no bits-per-value change: the scheme reduces WHICH positions are retained, fetched or reallocated, so it runs on whatever precision path the accelerator already has for the KV cache. A KV-position policy can never be a hardware format. Consequence for a reader: nothing further to verify - this is a cache-management policy, not a numeric format. The full statement of the three-way distinction, and the worked list of all 72 empty records by class, is in gotcha quant-native-support-empty-means-three-different-things. Rule of thumb for this field everywhere in the repo: if the bits field carries an 'a16', or the scheme is KV-cache-only or a llama.cpp integer storage type, an empty native_support is a DEFINITION and needs no verification; only a W4A4/W8A8/sub-8-bit format with research kernels can have an empty list that is a real finding.

Cross-references

Emulated support (5)