DeepSeek-V3 Technical Report

Frontier-scale MoE LLMs cost a great deal to train and to serve, and the two usual levers - more expert parallelism, more fine-grained routing - make training stability and inference communication harder.

A 671B-total / 37B-activated MoE trained on 14.8T tokens. (1) MLA from DeepSeek-V2 for the KV cache. (2) DeepSeekMoE with auxiliary-loss-free load balancing (a bias term per expert, rather than an auxiliary loss) plus a shared-expert route that is always on. (3) Multi-token prediction. (4) FP8 mixed-precision training across almost every matmul, with per-tensor and per-block scaling to preserve stability. (5) Dual-pipeline parallelism to overlap host-device communication with computation.

NOT AN INFERENCE SPEEDUP. The paper's headline operational claim is training cost: full training of DeepSeek-V3 required 2.788M H800 GPU hours. The abstract's other claims are quality ('outperforms other open-source models, comparable to leading closed-source models') and stability ('no irrecoverable loss spikes, no rollbacks'). There is no serving throughput or latency figure in the abstract. The serving-relevant numbers for this architecture are in the deepseek-v2-mla record instead.

arXiv ID 2412.19437 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. venue=arxiv-preprint: technical report, no venue claim in the arXiv comment, and an OpenReview search for the exact title returns no conference record. adoption=in-production rather than in-upstream-engine: this is the strongest adoption case in the batch - DeepSeek-V3 weights are released (github.com/deepseek-ai/DeepSeek-V3, 104k stars, pushed 2025-08-28) and both major engines serve them, so the method is deployed in production systems, not merely present in an engine's model zoo. Explicitly noted so the record is not read as claiming MLA belongs to this paper - MLA is DeepSeek-V2's contribution and is recorded there; what this report adds is MLA-in-a-shipped-frontier-model plus the FP8 training recipe. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/deepseek-v3-report.json and data/papers/paper-deepseek-v3.json were BOTH live records for arXiv 2412.19437 (DeepSeek-V3 Technical Report), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. paper-deepseek-v3.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2412.19437 with this title and the author list [DeepSeek-AI, Liu, Aixin, Feng, Bei, Xue, Bing], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "paper-deepseek-v3". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (951 chars vs 493) and is RETAINED here rather than discarded. Its text: 671B total, 37B activated per token (about 5.5 percent), MLA for the KV latent, FP8 mixed-precision training throughout, 14.8T training tokens, plus two novel mechanisms. (1) Auxiliary-loss-free load balancing: instead of adding a balancing term to the loss, add a bias b_i to each expert's routing logit and update it each step by b_i <- b_i + u * sign(f_i - P_i). Overloaded experts are pushed down and underloaded pushed up, and because the bias sits outside the network it cannot corrupt the gradient of the language-modelling objective, so balance is achieved with nothing to tune. (2) Multi-token prediction: an auxiliary self-prediction head per depth predicts t future tokens jointly, which both improves the model and leaves the MTP branches available at inference for SELF-SPECULATIVE DECODING, where the model drafts its own future tokens and verifies them in a single pass, trading extra compute per step for fewer sequential decode steps. speedup_reported: the two records carried DIFFERENT text for the same paper. This record's text is retained and the retired record's variant is quoted so the difference is visible: "2.788M H800 GPU-hours for the full 14.8T-token pre-training run. The report compares training cost against DeepSeek-V2 and Llama-3.1-405B in GPU-hours and explicitly declines to publish a self-reported cost multiple. The auxiliary-loss-free balancing win is reported as a small but consistent downstream gain over the auxiliary-loss variant rather than as a headline number. The MTP self-speculative-decoding benefit is described qualitatively in the report." hardware_relevance UNION taken: this record had H800, H100 SXM and H200 SXM; the retired twin had A100 80GB SXM4, H100 SXM and H200 SXM. H800 is the paper's own stated platform - 2.788M H800 GPU-hours - and A100 is added from the twin rather than dropped. AFFILIATIONS CORRECTED - arXiv API returns the collective author 'DeepSeek-AI' for 2412.19437; the retired twin's 'DeepSeek' is the same institution shortened. VENUE AND THE CLAIMS THAT COMPETED FOR IT: No peer-reviewed venue, agreed by both records: technical report, empty arXiv comment, and an OpenReview search for the exact title returns no conference record. The retired twin's `mechanism` was materially fuller - it gives the auxiliary-loss-free update rule b_i <- b_i + u * sign(f_i - P_i) explicitly - and is retained. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.