DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

MoE's training cost saving did not carry over to serving, because inference keeps all experts resident and pays an all-to-all every layer, so a dense model can be cheaper per token to serve.

End-to-end MoE training and inference system inside DeepSpeed with three separable contributions. (a) MoE architecture variants, including a DeepSeekMoE-style fine-grained expert split plus a shared expert that every token passes through, absorbing common knowledge and reducing routed load. (b) Model compression shrinking MoE models by up to 3.7x using expert pruning of below-average-importance experts alongside distillation, so the resident expert set shrinks rather than only the per-token compute; this is the genuinely novel and rarely copied part. (c) An inference kernel path that groups tokens into fixed-size buffers before dispatch, so the all-to-all becomes a few large bursty transfers instead of many small ones. Load balancing is the auxiliary-loss kind typical of this period.

7.3x better latency and cost than existing MoE inference solutions (the comparison set is prior MoE serving systems, not a dense model); MoE model size reduced by up to 3.7x via expert pruning plus distillation; and, for serving up to massive MoE models, up to 4.5x faster and 9x cheaper inference than quality-equivalent dense models. Also restates a 5x training cost saving for autoregressive LMs versus quality-equivalent dense models, which the paper attributes to shared work with GShard and Switch rather than to DeepSpeed-MoE alone. Measured on V100 and A100-class GPUs.

DeepSpeed ships the MoE layer and is a real upstream inference path, so the system is in-upstream-engine; the numbers themselves are DeepSpeed's own. Title text differs across arXiv versions: earlier versions read 'Advancing Mixture-of-Experts Inference and Training for Power-Scale Deep Learning'. The shared-expert idea from (a) became the standard recipe via DeepSeek; the expert-pruning-plus-distillation idea from (b) was largely not copied. VENUE CORRECTED 2026-10-04 (paper-verification pass): venue/year arxiv-preprint 2022 -> icml 2022. EVIDENCE: arXiv 2201.05596 comment field states verbatim: "This paper is published at ICML 2022: https://proceedings.mlr.press/v162/rajbhandari22a". PMLR v162 (ICML 2022, https://proceedings.mlr.press/v162/rajbhandari22a.html — the exact URL the comment field cites) contains it as rajbhandari22a. PREPRINT-vs-PUBLISHED conflation. The record holds the arXiv preprint number 2201.05596 AND asserts venue=arxiv-preprint, while its own arXiv comment claims ICML 2022 publication — a self-contradiction the record did not notice. The arxiv_id stays 2201.05596 (that is the preprint); the venue field now says icml so the record is marked peer-reviewed. Both values retained here deliberately; status set to contested because a peer-reviewed attribution changed. arxiv_id unchanged (2201.05596). VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. 2026-10-10: PMLR v162 ICML 2022 proceedings (idx1.py: https://proceedings.mlr.press/v162/ and pmlr.py: https://proceedings.mlr.press/v162/rajbhandari22a.html) confirms "DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale" (Rajbhandari et al., rajbhandari22a). arXiv API 2201.05596 (axc.py) comment states verbatim "This paper is published at ICML 2022: https://proceedings.mlr.press/v162/rajbhandari22a". Both sco-pmlr-icml-2022-v162 and arXiv comment agree on ICML 2022 (in venue enum). Status resolved to verified.