Serving DNNs like Clockwork: Performance Predictability from the Bottom Up

Reactive serving tricks (batching, warm pools) tame average latency but cannot bound the tail, because per-request execution time is not actually unpredictable — the internal state causing variance is just not modelled.

Refuse reactive control: predict the execution time of every request from its internal DNN state before it runs, using a predictor built from bottom-up execution analysis, then serve requests in a deterministic schedule that meets the latency target by construction. Tail latency stops being a scheduling problem.

NOT RECORDED as a multiple. The claim is qualitative and about tail behaviour: Clockwork reports meeting p99 latency targets by deterministic, predictor-driven scheduling where reactive batching baselines show large tails. Comparing it to any PagedAttention-era engine is anachronistic — this is a DNN convolution/resnet-era system whose variance sources are not the dynamic-batching and KV-cache effects that dominate LLM serving today.

Venue verified twice (arXiv comment naming OSDI '20, plus the OSDI '20 program page). Adoption research-only: the predictive-scheduling thesis was influential as an argument but no public engine implements Clockwork's model-based scheduling, largely because modern LLM serving's variance comes from dynamic batching and cache effects that a bottom-up predictor cannot see ahead of time. The DNN-era tail-latency framing also predates KV cache as the dominant memory consumer. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: osdi has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.