PackServe: SLO-Aware Request Scheduling for Agentic LLM Serving at Scale

Agentic workloads reuse KV cache across long shared prefixes, so naive request consolidation onto fewer instances trades away cache locality and prefill/decode interference, wrecking both SLO attainment and GPU efficiency at the same time.

Use compact white-box models to PREDICT latency under prefill/decode interference, then pack requests onto fewer serving instances subject to preserving KVC reuse and the token-level SLO — explicitly trading unused latency headroom for higher per-instance throughput. The white-box predictor is what makes safe packing possible without over-provisioning.

Per the arXiv abstract, on 64 NVIDIA H20 GPUs: up to 16.8% and 24.6% fewer GPU-hours than state-of-the-art schedulers under 30-ms and 50-ms TPOT targets respectively, while still meeting the target TPOT objectives. Note this is a COST/footprint reduction at equal latency, not a raw throughput multiple, and the hardware is H20 — a memory-capacity-and-bandwidth-constrained part, so read the result as evidence that scheduling efficiency can substitute for buying more accelerators.

Recorded adoption as 'unknown' rather than 'in-production' even though the abstract says the authors deployed it in their own environment: a paper's claim of internal deployment is not evidence a reader can use, and I could not verify a public repo or release. Venue deliberately 'arxiv-preprint' — the arXiv comment states only a page and figure count, with no venue. This is the clearest current example of the packing-vs-cache-locality tension that post-2024 agentic serving is fighting. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.