DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale

Inference at model scale needs kernels specialised per model family, and hand-writing them for every architecture does not scale as new transformer variants keep appearing.

A kernel JIT library that compiles a specialised inference kernel per model automatically, plus the multi-GPU inference engine built on ZeRO-Inference-style sharding and parameter offloading, so a model far larger than one device's memory can be served with kernels tuned to it.

NOT RECORDED as a bare multiple. The paper reports large throughput improvements for OPT and other transformer families from its automatic kernel generation, but I did not verify the exact figures with their baselines and settings, and the gain is strongly per-model-family. Also distinguish from the companion arXiv 2401.08671 (DeepSpeed-FastGen), whose abstract does give a figure versus vLLM (2.3x effective throughput).

Venue verified via Crossref: SC22 (DOI 10.1109/SC41404.2022.00051). Recorded as 'other' because SC/HPC venues are not in the schema enum — the SC22 attribution is here so it is not lost. Adoption in-production: DeepSpeed remains an actively used Microsoft library and MII/DeepSpeed-Inference is deployed, though it has been largely superseded for mainstream LLM serving by vLLM/SGLang/TensorRT-LLM. The kernel-JIT idea is the durable part. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. SC22 (IEEE/ACM SC) has no Findings or workshop tier for this track; the record already documents the SC22 attribution in notes with DOI 10.1109/SC41404.2022.00051. VENUE RECLASSIFIED 2026-10-10 (crawler/w11-venue-enum): venue 'other' -> 'sc22', venue_kind 'conference'. Confirmed by record notes and agreeing publisher/index deposit in data/sources/.