DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

For autoregressive decode the KV cache is the memory bottleneck, and it gets worse with context length, batch size and every KV head - GQA reduces it only by an integer factor while the cache still scales linearly in sequence length.

Multi-head Latent Attention (MLA). Down-project the concatenated per-head K and V into a single low-dimensional latent vector (much smaller than the K+V total), CACHE ONLY THE LATENT, and up-project back to per-head K and V at attention time. Absorption of the up-projection into the query lets decode-time attention read the small latent directly without materialising per-head K/V, so the cache is compressed by the projection rank rather than by dropping heads. Plus DeepSeekMoE: fine-grained expert segmentation plus shared experts, and load-balancing auxiliary losses.

Paper's own claims (abstract), all vs DeepSeek 67B: saves 42.5% of training cost, reduces the KV cache by 93.3%, and boosts maximum generation throughput by 5.76x. These are three different quantities (cost, memory, throughput) against one named baseline - not interchangeable, and the KV-cache and throughput numbers are the two that matter for serving. Trained on 8.1T tokens with 21B activated parameters.

arXiv ID 2405.04434 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. venue=arxiv-preprint is CORRECT here, not a shortfall: DeepSeek-V2 was a technical report and OpenReview returns no conference record for it (a search for the exact title returns only DBLP 'CoRR 2024'). Per the rule that comment-field-only evidence yields arxiv-preprint, and here there is no venue claim at all. adoption=in-upstream-engine verified in source: vllm/model_executor/models/deepseek_v2.py (79 KB, 16 MLA references) is the shipped vLLM path for DeepSeek-V2 and -V3, and SGLang ships flashmla_backend.py, deepseek_v4_backend.py and tokenspeed_mla_backend.py. The mechanism text is the load-bearing part: 'cache a low-rank latent and absorb the up-projection into the query' is precisely what makes the 93.3% number achievable, and it is the distinction from GQA that a reader needs. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/deepseek-v2-mla.json and data/papers/paper-deepseek-v2.json were BOTH live records for arXiv 2405.04434 (DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. paper-deepseek-v2.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2405.04434 with this title and the author list [DeepSeek-AI, Liu, Aixin, Feng, Bei, Wang, Bin], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "paper-deepseek-v2". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (803 chars vs 573) and is RETAINED here rather than discarded. Its text: Combines two independent mechanisms. (1) DeepSeekMoE: fine-grained mN experts with mK activated plus K_s always-on shared experts; 236B total, 21B activated per token, 128K context. (2) Multi-head Latent Attention (MLA): instead of caching K and V per KV head, project the hidden state once per token down to a low-dimensional latent, cache ONLY that latent, then up-project to per-head K and V at attention time, with decoupled RoPE applied to a separate small vector so positional information survives the down-projection. Per-token KV cache becomes O(d_c) instead of O(L * d_h * d_k) and, crucially, INDEPENDENT of the number of KV heads, so raising head count for quality no longer multiplies cache size. MLA is the mechanism that changes the serving memory bottleneck; the MoE block alone does not. speedup_reported: the two records carried DIFFERENT text for the same paper. This record's text is retained and the retired record's variant is quoted so the difference is visible: "Saves 42.5 percent of training cost relative to DeepSeek 67B while substantially improving quality (21B versus 67B active parameters). The inference-side MLA claim is reported as a KV-cache size reduction rather than a tokens/second number, so there is no serving throughput figure to quote. Reported on H800 GPUs, a China-market SKU close to H100 SXM compute." AFFILIATIONS CORRECTED - arXiv API returns the collective author 'DeepSeek-AI' for 2405.04434; the retired twin's plain 'DeepSeek' is the same institution under a shorter rendering, and the form matching the paper's own byline is kept. VENUE AND THE CLAIMS THAT COMPETED FOR IT: No peer-reviewed venue, agreed by both records, and here the NEGATIVE is the finding rather than a gap: DeepSeek-V2 was a technical report, the arXiv comment is empty, and an OpenReview search for the exact title returns only the DBLP CoRR deposit. Both records independently reached the same value. The one substantive prose difference is preserved rather than resolved: the retired twin's `mechanism` was materially fuller and correctly separates DeepSeekMoE (fine-grained mN experts with always-on shared experts) from MLA, which is the distinction this record's narrower 'kv-cache' framing loses. The retired twin's category 'moe' versus this record's 'kv-cache' is a framing choice about which of the paper's two separable contributions leads; both are defensible and kv-cache is kept because MLA is the contribution that changed serving. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.