Design goal
goodput-under-TTFT-and-TPOT-SLOslow-latencyper-phase resource allocation
Scheduling
continuous-batchingpaged-kvdisaggregated-prefill-decodeper-phase parallelism planning
Notable features
Prefill and decode assigned to different GPUs to eliminate prefill-decode interference, with resource allocation and parallelism strategy co-optimised per phase against TTFT and TPOT targetsPhase placement is bandwidth-aware: the paper places the two phases 'according to the serving cluster's bandwidth to minimize the communication caused by disaggregation', which is the first explicit treatment of placement as part of the algorithmAutomatic KV-cache communication and memory management between the two instances - the README says it 'works just like a single instance'Built on a custom C++ execution backend (SwiftTransformer) with model/pipeline parallelism, FlashAttention, continuous batching and PagedAttentionRay-based distributed workers
Hardware caveats
Research artifact, not maintained: repo created 2024-01-16, last push 2025-04-06, ~839 starsRequires its own custom C++ backend (SwiftTransformer) plus a Ray cluster - it is not a drop-in on vLLM or SGLangModel coverage is GPT-2, OPT and LLaMA2 only, per the README - no modern architectures, no MoE, no long-context variantsCUDA only
Best for
Studying the disaggregation cost model: the paper is the cleanest statement of why colocating the phases couples two different bottlenecks, and of why placement must be bandwidth-awareReading rather than deploying - the OSDI 2024 goodput framing (max request rate under joint TTFT and TPOT constraints) is the right way to evaluate any disaggregation claim
Avoid for
Production deployment - unmaintained since April 2025, narrow model support, custom backend requiredAnything needing MoE, FP8/FP4 or long contextComparing against a modern stack: its baseline is Orca/vLLM-0.4-era, so speedup ratios do not transfer
notes
Research prototype (OSDI 2024), not a production system - recorded as the paper that named the problem. KV transfer path: inter-GPU KV-cache communication whose placement is explicitly a function of the cluster's interconnect bandwidth; the abstract states the placement step and the paper is the primary source for 'disaggregation cost is bytes x bandwidth'. Reported result: '7.4x more requests or 12.6x tighter SLO, compared to state-of-the-art systems, while staying within latency constraints for > 90% of requests'. Read this as a goodput metric under joint TTFT/TPOT SLOs, not a throughput multiplier - the two are routinely conflated and vLLM's own documentation takes the opposing position on throughput. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2025-04-06 (17.9 months before this record), NO GitHub release or tag found, 6 contributors (distinct committers including anonymous), open_issues_count 35 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 839 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.