Fast Transformer Decoding: One Write-Head is All You Need

Autoregressive decoding of a transformer is memory-bandwidth-bound, not compute-bound, because every step re-reads the large key and value tensors from HBM - and those tensors grow with the number of attention heads.

Multi-query attention: share ONE key head and ONE value head across all query heads, so the per-step KV read shrinks by the number of attention heads. Training is unchanged in cost (the attention computation is unchanged in shape); only the decode-time memory traffic drops. Squeezes the KV cache from H heads to 1.

QUALITATIVE ONLY, and the paper is careful about this: 'we verify experimentally that the resulting models can indeed be MUCH FASTER to decode, and incur only minor quality degradation from the baseline'. No multiple, no baseline engine, no hardware. The mechanism argument is the durable claim: decode is bandwidth-bound, so shrinking the KV tensor by the head count is a direct bandwidth reduction. Any specific decode-tok/s figure attributed to this paper would be borrowed from a later implementation.

arXiv ID 1911.02150 verified via abs-page meta tags on 2026-10-03; title matches. venue=arxiv-preprint is deliberate and matches the dead agent's source note: a later venue version exists (the paper was published at ICLR 2020 and is often cited as such) but was NOT confirmed from any primary source this session - OpenReview search returns no matching record and Semantic Scholar was 429 throughout. Per the rules, unverified venue means arxiv-preprint plus this note, not an assumed ICLR. adoption=in-upstream-engine: MQA is a single-KV-head model config; vLLM and SGLang serve such models through the generic attention path, and MQA is the limiting case of the GQA family (grouped-query-attention) that shipped afterwards. No kernel work was needed, which is exactly why adoption is total for the idea and zero for any code. DANGLING REF, LEFT UNRESOLVED (2026-10-04 cross-reference repair): hardware_relevance carries 'nvidia-tesla-p100', which is not a record id and has no target - data/accelerators/ contains no P100 record at all. The sibling values in this field (nvidia-v100-sxm2-32gb, nvidia-a100-40gb-sxm4, nvidia-a100-80gb-sxm4) all resolve, so the field is otherwise sound. Left unchanged; the honest options are adding the missing accelerator record or dropping the value. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.