id
gotchas/ops-kv-cache-full-preempts-and-recomputes
Concepts
kv-cachecontinuous-batchingpreemptionchunked-prefillprefix-cachingpaged-kv
Symptom
Latency climbs sharply for a subset of requests — typically the ones with long inputs and short outputs — while error rate stays at zero and the request is still 'served'. vLLM's log line shows a rising 'Preemptions: N' count; SGLang logs retractions. The affected requests have already streamed or will re-run their entire prompt, and their time-to-first-token is multiplied by a re-prefill they never asked for. Under a burst this can oscillate: preemption frees blocks, the freed requests re-enter and are preempted again.
Root cause
vLLM's _preempt_request asserts the request is RUNNING, frees its KV blocks, sets status to PREEMPTED, and — this is the load-bearing line — sets 'request.num_computed_tokens = 0' before prepending it back to the waiting queue. Every computed token is discarded, so the request must re-prefill its whole prompt. The victim is popped from the end of the running queue (LIFO) unless SchedulingPolicy.PRIORITY is set; default FIFO policies therefore evict the most recently started request, not the oldest. SGLang exposes the same class of behaviour as --retraction-policy, documented as 'The decode retraction policy to use when the KV cache is full. "length" preserves the existing behavior and retracts short-output, long-input requests first' — so the default victim selection is explicitly hostile to long-prompt requests. vllm:num_preemptions is the counter to watch ('Cumulative number of preemption from the engine').
Workaround
Alert on vllm:num_preemptions (and SGLang's retraction behaviour) before latency, since preemption precedes the latency cliff. The mitigations are capacity or policy, not tuning: bound max_model_len so a single request cannot claim the cache, use --max-num-seqs to cap concurrency, enable chunked prefill so a long prompt cannot monopolise a step, and if you use priorities enable --enable-priority-scheduling with a non-default --retraction-policy so victims are chosen by priority rather than by length. SGLang's own advice for a different symptom of the same cause is '--schedule-conservativeness ... Use a larger value if you see requests being retracted frequently.' Note that preemption is also reachable administratively: reset_prefix_cache(reset_running_requests=True) drains the entire running queue through the identical preemption path.
notes
Silent by construction: a preempted request is not an error, so a 0% error rate coexists with severe tail latency. This is the specific reason a request-count-based SLO will pass while users are unhappy. Partial mitigation only — preemption cannot be eliminated, only made rarer and more fairly targeted, because the cache is genuinely finite. SCHEME RECLASSIFIED (2026-10-04): was 'config'. RE-VERIFIED 2026-10-05 against vLLM v0.30.0 (release 2026-09-22; main commit 2026-10-05T04:19:19Z). STILL HOLDS, with the entrypoint tree now refactored under vllm/entrypoints/launchers/ and vllm/entrypoints/openai/api_server.py emitting a DeprecationWarning. Scheduler._preempt_request() in vllm/v1/core/sched/scheduler.py still contains the load-bearing line 'request.num_computed_tokens = 0' immediately after setting status to PREEMPTED, still calls _free_request_blocks, and still prepends the request back onto the waiting queue. The only new state is request.drop_stale_output / num_stale_output_tokens bookkeeping for async scheduling, which affects how in-flight output is delivered after a resume - not whether the prefill is redone from zero.