DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

GShard-style top-k-of-N MoE keeps experts overlapping and redundant, so DeepSeekMoE 16B merely matched a 2.9B GShard and still lost to a same-size dense model instead of beating it.

Two changes to the MoE block. (1) Fine-grained experts: split N experts into mN experts of intermediate size 1/m each and activate mK of them instead of K of N, which decouples expert granularity from activation count and enlarges the space of expert combinations at identical FLOPs. (2) Shared experts: isolate K_s experts that every token always passes through and that receive no routed tokens, so they absorb common knowledge, reduce redundancy among routed experts, and let the router specialise. Every token therefore activates mK routed experts plus K_s shared experts. Routing is token-choice top-k with an auxiliary load-balance loss, the same family as Switch and GShard; it is the shared-expert addition that makes the fine-grained split pay off. Shipped ratios: DeepSeek-V2 236B total / 21B active (about 8.9 percent); DeepSeek-V3 671B total / 37B active (about 5.5 percent).

At 2B scale and equal training compute, DeepSeekMoE 2B matches GShard 2.9B even though the GShard baseline has 1.5x the expert parameters and 1.5x the expert FLOPs. At 16B scale, DeepSeekMoE 16B matches LLaMA2 7B using only about 40 percent of its FLOPs. At 145B scale, DeepSeekMoE 145B matches DeepSeek 67B using only 28.5 percent of the computation, or 18.2 percent against the dense bound it is extrapolated from. All are quality-matched FLOP comparisons on H800-class GPUs, not throughput measurements.

Load balancing here is the AUXILIARY-LOSS kind inherited from GShard. DeepSeek-V3 later replaced it with the auxiliary-loss-free bias term, so citing DeepSeekMoE as the source of load balancing is wrong; cite DeepSeek-V3 for that. The shared-expert plus fine-grained-split recipe is now the de facto default for new open MoE models and is the most copied architectural contribution in this slice. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass): venue_track set to 'main' from the ACL Anthology bib index (anthology.bib, 131650 entries) by normalized title match. Evidence: anthology id 2024.acl-long.70, booktitle 'Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. The Anthology id is the authoritative signal here and it is unambiguous, so this is an additive clarification of a track, not a change of venue - `venue` is untouched and no status change is implied. The reason this matters for inference readers: a Findings paper and a main-track paper at the same venue are not the same kind of evidence, and until this field existed the repo could not say which was which. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'acl', venue_track 'main'. EVIDENCE: the ACL Anthology bib index (anthology.bib, 131650 entries, fetched 2026-10-04) contains this exact title as anthology id 2024.acl-long.70 with booktitle 'Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. That is a proceedings record, so 'no venue found' was an under-claim, not a fact. BOTH VALUES RETAINED HERE PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2401.06066). 2026-10-10: arXiv API 2401.06066 (axc.py: http://export.arxiv.org/api/query?id_list=2401.06066&max_results=1) returns comment=null. ACL Anthology 2024.acl-long.70 (https://aclanthology.org/2024.acl-long.70/) confirms "DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models", ACL 2024, pages 1280-1297. Source pap-arch-2401-06066 notes cite arXiv preprint while ACL Anthology confirms ACL 2024. Sources disagree; status remains contested. 2026-10-10: Crossref deposit fetched via https://api.crossref.org/works/10.18653/v1/2024.acl-long.70 returned title 'DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models' matching container 'Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)'. Status resolved to verified with venue acl (conference).