DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

Colocating prefill and decode in one batch couples their resource-allocation and parallelism plans, so a serving system must trade TTFT against inter-token latency instead of meeting both SLOs.

Place prefill workers and decode workers on disjoint GPUs so the two phases cannot preempt each other, then independently choose tensor/pipeline parallelism and worker count per phase to hit a TTFT target and a TPOT target separately. Phase placement is chosen against measured cluster bandwidth to keep the KV handoff cost low. Scheduling uses a parallelism-agnostic goodput-maximising policy that treats each request as a two-stage job.

Per the OSDI '24 proceedings abstract: 7.4x more requests served OR 12.6x tighter SLO versus state-of-the-art LLM serving systems, while staying within latency constraints for >90% of requests, across popular LLMs, applications and latency requirements. The metric is goodput — the maximum request rate served within BOTH TTFT and TPOT constraints on each GPU. The two figures are alternative operating points, not multiplicative.

SLUG RENAMED 2026-10-04 (parent agent): was 'distserve', colliding with the engines record of the same id, which made every bare-slug cross-reference ambiguous. Inbound refs ([]) all mean the engines record, so none needed repointing; papers are cited via `sources`, never by bare slug. Venue verified twice (arXiv comment 'OSDI 2024' plus the OSDI '24 presentation page). Adoption is honestly research-only: DistServe's own prototype was never released as a usable system, and no production inference engine is built on it. Its IDEAS are in production — prefill/decode disaggregation is now a shipped vLLM feature — but vLLM's implementation is not DistServe's code. Cite this for the architectural argument, not as software you can install. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: osdi has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.