Kairos: P95 TTFT breakdown in disaggregated serving — prefill execution is 2-23%, queue+KV transfer is 77-98%

Hardware: 2 prefill + 2 decode nodes (2P2D), each NVIDIA A100 80GB. Engine: vLLM 0.18.1 with Kairos prefill-deflection scheduler. Model: DeepSeek-V2-Lite (MoE). Workload: production-derived trace normalized to 32K context. Five workloads tested: LPLD (low prompt/low decode), LPHD, HPLD, HPHD, Bursty. Cache state: cold (no prefix caching mentioned). P95 TTFT breakdown for LPLD: prefill queue wait = 59.2 ms, prefill execution = 191.4 ms, KV-cache transfer = 4205.4 ms, decode queue wait = 54.5 ms, end-to-end TTFT = 6278.8 ms. Prefill execution is only 3.1% of P95 TTFT for LPLD. For HPLD: prefill execution = 1340.2 ms (23.2% of 5781.5 ms). For HPHD: prefill execution = 1222.8 ms (19.4% of 6292.0 ms). For Bursty: prefill execution = 1070.1 ms (13.6% of 7893.9 ms).

Published on arXiv:2607.02043 (2026-07-02) by Microsoft Research. The most important TTFT breakdown finding for disaggregated serving: prefill execution accounts for only 2-23% of P95 TTFT, while prefill queue wait and inter-node KV-cache transfer account for 77-98%. This means that in disaggregated setups, the actual prefill computation is NOT the bottleneck — scheduling and data movement are. Deflecting prefill to decode nodes (building KV cache in place) eliminates KV transfer and reduces P95 TTFT by up to 81%. Also shows decode nodes are severely underutilized (SM utilization 7-27% vs 75-84% on prefill nodes). LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (4 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - engine version: unrecoverable from paper-kairos-prefill-deflection-ttft-breakdown - engine_id=vllm is named but no build/commit/container digest is published - batch/concurrency: unrecoverable from paper-kairos-prefill-deflection-ttft-breakdown - would need the achieved in-flight batch or the --max-concurrency value - output length: unrecoverable from paper-kairos-prefill-deflection-ttft-breakdown - would need the run's OSL or the harness's configured max_tokens - warmup: unrecoverable from paper-kairos-prefill-deflection-ttft-breakdown - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted.