AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

One-model-per-GPU serving wastes each GPU's memory bandwidth whenever a single model cannot saturate it, and high-bandwidth memory designs are wasted on models too small to fill them.

Repurpose model parallelism as statistical multiplexing: split several small models across a group of devices with pipeline parallelism, accepting the pipeline overhead in exchange for better bandwidth utilisation. Placement follows a new scaling argument relating model size to device count rather than to memory capacity alone.

NOT RECORDED as a multiple — the arXiv abstract describes the statistical-multiplexing argument but I did not verify a specific throughput figure with its baseline, model mix and device count. The paper is OSDI '23; read its evaluation section (A100/V100 clusters, multi-model serving) before citing a speedup. The durable claim is the scaling insight — that model parallelism can multiplex models even when one model fits on one device — not any particular number.

Venue verified twice (arXiv comment 'OSDI 2023' plus the OSDI '23 program page). Adoption research-only: the idea that pipeline parallelism can multiplex models is real and echoed in later serving work, but AlpaServe itself was not productised and no engine ships it. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: osdi has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.