Decomposing Predictive Kubernetes Autoscaling for Large Language Model Serving Under Long Startup Delays

New serving replicas take 2-10 MINUTES to start because multi-gigabyte weights must load, which makes purely reactive autoscaling structurally late — no amount of reaction speed can beat the actuation delay.

Decompose predictive autoscaling into four factors — token-aware demand tracking, startup-delay lookahead, a bounded uncertainty margin, and plant-state observation — and measure each factor's contribution in isolation on production-derived heavy-tailed workloads from ServeGen. Conclusion: a simple exponentially weighted moving average with delay-aware lookahead and an upper-confidence-bound margin captures most of the benefit.

Per the arXiv abstract: reduces TTFT SLO violations from 53% (reactive, queries-per-second based) to 0.5% across five random seeds; lookahead alone is the single largest factor at a 14x reduction. Explicit negative result also reported: Kalman-filter variants do NOT consistently beat the EWMA. This is a usefulness result about scheduler DEPLOYMENT rather than request-level throughput — the baseline is reactive QPS-based scaling, which is the weak baseline it should be compared against.

VENUE CAVEAT, stated plainly: the arXiv comment field says 'accepted by CloudCom 2026' and that is my ONLY evidence — I did not confirm it against a CloudCom proceedings page, so the venue is recorded as arxiv-preprint and CloudCom is not in the schema enum regardless. This is the scheduling-plane counterpart to the data-plane scheduling papers above and covers the layer most serving papers ignore: adding capacity. The negative result (Kalman filters lose to EWMA) is the most transferable finding. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.