vLLM PR #51281: optimize DecodeStream priming with a bounded prompt tail
Fields
sources/flop3-vllm-pr-51281-bounded-tail-detokNotes
The one-time-per-request detokenisation cost that lands in TTFT. FastIncrementalDetokenizer primed its DecodeStream with the ENTIRE prompt, and the native prefill does O(prompt_len) work: ~0.5-1.5 us/token measured with tokenizers 0.22.2 (whose Rust binding defers the decode to the first step() call), i.e. ~3 ms for 5K ids, ~9 ms for 15K, ~12 ms for 20K. It runs on the API-server event loop, so it lands verbatim in that request's streaming TTFT and head-of-line blocks the SSE deltas of every other active stream on the same loop (p99 contamination). Fix: prime with the last 32 prompt ids (DETOKENIZATION_OFFSET), decoding once per request in ~7 us. Precedent cited in the PR: vLLM's slow path has always primed with the last 7 ids, and SGLang with the last 5.
Cross-references
This record references no other records.