id
gotchas/ops-vllm-admission-rejects-503-not-429
Concepts
kv-cachequeueingadmission-controlratelimitcontinuous-batchingopenai-api
Symptom
A load test or client with a 'retry on 429' rule reports 100% success while the server is shedding a large fraction of requests as HTTP 503 Service Unavailable. The 503s appear in the vLLM access log and as elevated 5xx, so an SLO built on 429 rate will read zero throttle events and conclude the deployment is healthy. Operators comparing 'requests in' against 'vllm:request_success' see a permanent gap that looks like a metrics bug.
Root cause
AsyncLLM.check_admission() raises QueueOverflowError (max_num_queued_reqs) or MaxQueuedTokensError (max_num_queued_tokens), and its own docstring states 'Both limits return HTTP 503 (Service Unavailable) so that load balancers and client SDKs retry on a different instance.' The design intent is deliberate — 503 is a 'try another instance' signal, whereas 429 conventionally means 'you are being rate limited, back off'. But it means the throttle semantics are invisible to any client, dashboard or alerting rule written against the OpenAI convention of 429 for overload, and vLLM does not emit a distinct metric or header distinguishing an admission rejection from an internal error. Both rejection paths log at logger.info ('Request queue full - rejecting request ...' / 'Max queued tokens reached - rejecting request ...'), so they are invisible at default log level.
Workaround
Treat 503 from a vLLM OpenAI-compatible endpoint as a throttle signal: alert on the rate of 5xx from the completion endpoints, and instrument the admission path yourself. Set explicit queue bounds (--max-num-queued-reqs, --max-num-queued-tokens) so shedding is a deliberate, bounded policy rather than whatever the client timeout imposes. Note that the two rejections are not equally recoverable: max_num_queued_tokens is a TTFT guard on *prefill* tokens and fires early and conservatively by design; max_num_queued_reqs counts waiting+running requests and is the request-count guard. Contrast with TGI, which returns a genuine 429 for the same class of condition (see ops-tgi-429-is-semaphore-not-queue), and with sgl-router, which returns 429 for load shedding and 503 for draining.
notes
Both bounds default to None (off) — see ops-vllm-engine-arg-defaults — so on a stock deployment there is no admission rejection at all; the request simply enters the waiting queue and the client times out instead. The silent mode is worse than the loud mode, so this gotcha only bites operators who have correctly turned the bounds on. The overcount is deliberate and documented: get_num_queued_tokens uses prompt_len for ALL prefilling requests without subtracting chunked-prefill progress or prefix-cache hits, because the scheduler only propagates num_computed_tokens and num_cached_tokens to the API server after prefill completes; the code calls this 'conservative — earlier rejection, preserving TTFT targets'. The operational consequence is that a prefix-cache-heavy workload is rejected earlier than its true token cost warrants. SCHEME RECLASSIFIED (2026-10-04): was 'config'. RE-VERIFIED 2026-10-05 against vLLM v0.30.0 (release 2026-09-22; main commit 2026-10-05T04:19:19Z). STILL HOLDS, with the entrypoint tree now refactored under vllm/entrypoints/launchers/ and vllm/entrypoints/openai/api_server.py emitting a DeprecationWarning. The status code is now traceable end to end rather than only documented: AsyncLLM.check_admission() in vllm/v1/engine/async_llm.py still raises QueueOverflowError / MaxQueuedTokensError and its docstring still reads 'Both limits return HTTP 503 (Service Unavailable) so that load balancers and client SDKs retry on a different instance', but the binding now lives in vllm/exceptions.py, where both classes subclass GracefulHTTPError and pass HTTPStatus.SERVICE_UNAVAILABLE to super().__init__(). No 429 path was found anywhere in vllm/v1/engine/ or vllm/entrypoints/. The gotcha is therefore still current, and if anything stronger: the 503 is now a constructor argument someone could change, not just prose.