id
gotchas/ops-restart-aborts-in-flight-by-default
Concepts
graceful-shutdownkubernetespreemptionrolling-updateblue-greencanary
Symptom
Every rolling update or scale-down fails a batch of in-flight requests. The failures land as error-status responses to clients mid-generation, typically on the pods being replaced, and cluster-level error rates spike in proportion to in-flight concurrency at the moment of termination. If requests are long-running, the failures are worse than the spike suggests: a request that had already produced most of its output is thrown away entirely.
Root cause
shutdown_timeout defaults to 0 (see ops-vllm-engine-arg-defaults), and vLLM's _handle_shutdown branches on exactly that value: mode = 'abort' if shutdown_timeout == 0 else 'drain'. With 0 it logs '[shutdown] EngineCore: aborting in-flight requests count=%d' and calls scheduler.finish_requests(None, RequestStatus.FINISHED_ABORTED), sending abort outputs for every one. llm-d states the operator-facing consequence plainly: 'Default - vLLM immediately aborts in-flight requests and exits. Those requests fail with an error status code.' Kubernetes makes this the default outcome of routine operations — scale-down, rolling update, node drain — and the pod termination sequence is Terminating, endpoint removal, preStop, SIGTERM, then terminationGracePeriodSeconds (default 30s) before SIGKILL. So a stock deployment has both the abort behaviour and, independently, a grace period too short for any real drain.
Workaround
Set --shutdown-timeout N on every vLLM replica, sized as llm-d advises: 'Set --shutdown-timeout to roughly the p99 request duration you want to allow to complete on drain.' Then the critical, easily-missed constraint: 'terminationGracePeriodSeconds must be greater than --shutdown-timeout (plus a small buffer for the preStop hook and process cleanup). Otherwise Kubernetes sends SIGKILL before the drain completes, and the graceful window is never fully used.' Their recommended shape is terminationGracePeriodSeconds 120 with --shutdown-timeout 90. Two related settings: pair it with a model-aware readiness probe so the endpoint is removed from the pool before SIGTERM arrives (endpoint removal happens on readiness, not on signal), and make the routing layer's queued-request behaviour retryable — llm-d's EPP evicts queued requests with a retryable 503 (outcome rejected-shutting-down) rather than a hard fault, but only 'while the EPP is still reachable over its ext_proc stream'.
notes
The trap is the ordering: setting --shutdown-timeout without raising terminationGracePeriodSeconds looks configured and behaves exactly like no configuration at all, because SIGKILL preempts the drain. Verify by actually draining: start a long generation, delete the pod, and confirm the request completes and the pod exits before the grace period elapses. Also note that client disconnect reclaims resources promptly (vLLM's abort codepath frees the KV cache on disconnect), so a gateway that retries by disconnecting the original request lets a draining pod release capacity faster than waiting for it. SCHEME RECLASSIFIED (2026-10-04): was 'config'. RE-VERIFIED 2026-10-05 against vLLM v0.30.0 (release 2026-09-22; main commit 2026-10-05T04:19:19Z). STILL HOLDS, with the entrypoint tree now refactored under vllm/entrypoints/launchers/ and vllm/entrypoints/openai/api_server.py emitting a DeprecationWarning. EngineCore.shutdown() in vllm/v1/engine/core.py still branches on the value verbatim - 'mode = "abort" if shutdown_timeout == 0 else "drain"' - and the default is still literally 'shutdown_timeout: int = 0' in vllm/engine/arg_utils.py (line 834). The abort path still calls scheduler.finish_requests(None, RequestStatus.FINISHED_ABORTED) and logs '[shutdown] EngineCore: aborting in-flight requests count=%d'. Unchanged.