id
papers/llumnix-dynamic-scheduling
Authors
Biao SunZiming HuangHanyu ZhaoWencong XiaoXinyi ZhangYong LiWei Lin
Affiliations
Alibaba Group
Problem
LLM requests are heterogeneous and unpredictable in both resource demand and latency requirement, so a static batching or partitioning policy produces severe queueing delay, poor tail latency and SLO violations when a single high-priority or long request stalls a shared batch.
Mechanism
A general LLM serving system built on a flexible abstraction: requests are abstracted to an execution-agnostic interface so policies (migrating live requests between instances, swapping between continuous batching and request-level batching, disaggregated vs aggregated execution) can be written once and composed, rather than being hard-coded per system. Scheduling decisions are driven by per-step resource profiling and priority.
Speedup reported
Paper's own claim: reduces queueing delay and improves throughput and tail latency versus baselines including Orca, vLLM and a prefill-decode-disaggregated variant, under mixed-priority workloads. Numbers are workload-specific; the durable claim is that the abstraction lets one system match whichever scheduling policy suits the load.
notes
Added because the repo's scheduling records are all from one design axis (chunked prefill in sarathi-serve, multiplexed parallelism in alpaserve/muxserve, disaggregation in distserve) and none covers the cross-instance MIGRATION axis, which is what Llumnix contributes: moving a partially-served request to another instance mid-flight. Venue OSDI 2024. arXiv comment reads "To appear at OSDI '24" and Semantic Scholar independently returns 'USENIX Symposium on Operating Systems Design and Implementation' 2024. The OSDI'24 USENIX presentation page was not separately fetched, so this rests on two agreeing sources rather than the proceedings listing. Affiliation 'Alibaba Group' is stated directly on the PDF title block and corroborated by OpenAlex.