DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference

Existing frameworks force a choice between throughput and latency for LLM generation, and this hurts most with long prompts where the prefill phase dominates.

Dynamic SplitFuse: decompose each request into prompt and generation work, then compose the split so prefill for the next request overlaps generation of the current one, with the split point chosen dynamically per request rather than fixed.

Per the arXiv abstract, versus state-of-the-art systems LIKE vLLM: up to 2.3x higher effective throughput, 2x lower average latency, and up to 3.7x lower token-level tail latency, for long-prompt generation workloads. The system is built by combining DeepSpeed-MII with DeepSpeed-Inference. Reported across various models and hardware configurations; read the paper's benchmarking methodology section for the exact settings, since long-prompt effective throughput is highly sensitive to prompt length.

VENUE NOT VERIFIED — arXiv comment is empty and I found no proceedings record; recorded as arxiv-preprint deliberately. Adoption in-production but with a caveat worth stating: FastGen ships as a mode in DeepSpeed-MII, which is deployed inside Azure and other Microsoft stacks, yet DeepSpeed's serving path has clearly lost ground to vLLM/SGLang in the open-source ecosystem. Dynamic SplitFuse is the same family of idea as Sarathi-Serve chunked prefill and was developed independently. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.