MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving

When LLM popularity is highly skewed, dedicating whole GPUs per model wastes most of each GPU — the hot model needs only part of a GPU while the tail model sits on an under-used device.

Spatial-temporal multiplexing: colocate models by co-locating the KV cache of an under-loaded model into the leftover GPU memory of a hot model's device (spatial), then share the compute of those devices across time (temporal) with per-model batching degrees chosen by an optimisation over the load distribution.

Per the arXiv abstract: up to 1.8x higher throughput, OR 2.9x more requests processed within 99% SLO attainment, against the paper's baselines (single-model-per-GPU and prior multiplexing designs). These are alternative operating points, not additive — the throughput and SLO-attainment figures describe the same system at different settings. Gain is entirely a function of how skewed model popularity is.

Venue verified on OpenReview ('ICML 2024' / 'ICML 2024 Poster'); note the OpenReview title differs slightly ('Flexible Multiplexing for Efficient Multiple LLM Serving'). Adoption research-only: MuxServe is NOT the origin of SGLang's RadixAttention despite the superficial similarity (both do prefix/KV reuse across a device boundary) — conflating them is an easy and wrong citation. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.