llm-d EPP Flow Control reference (centralised queuing, saturation detectors, eviction, error mapping, what it does not solve)

Behind the flowControl feature gate. The RPS argument, verbatim: 'Traditional API gateways rate-limit by Requests Per Second (RPS), assuming every request consumes roughly the same resources. In LLM serving, this approach is ill-suited. Resource consumption is driven by long input contexts and the unpredictable autoregressive decode loop (output tokens). A single request can consume orders of magnitude more compute and memory than another ... Static RPS limits either starve the pool by being too conservative or cause overload by being too aggressive.' Named failure modes without Flow Control: The Noisy Neighbor ('A single tenant sending massive prompts can monopolize an endpoint KV cache and queue, starving others'), Scheduling Regret ('Once a request is dispatched to a model server local queue, the EPP cannot move it. Premature routing locks requests into suboptimal candidates'), Priority Inversion ('Model servers batch based on arrival or sequence length, not business priority. Critical real-time requests can get stuck behind large offline batches'), Resource Asymmetry ('A single request with a massive context can consume more resources than hundreds of short requests'). Limits are global maxBytes and maxRequests plus per-band maxBytes/maxRequests giving 'memory isolation between bands; heavy traffic in a lower-priority band cannot fill the queue space reserved for higher-priority bands'; exceeding them is an immediate HTTP 429, and the doc distinguishes proxy-side maxBytes/maxRequests (gateway memory) from the Saturation Detector (GPU compute and KV capacity). Fairness policies: global-strict-fairness-policy 'Ignores flow isolation and serves all requests in a single global order' and round-robin-fairness-policy which 'Prevents a single high-volume flow from starving others'. Ordering: fcfs (default), edf, slo-deadline-ordering-policy reading x-llm-d-slo-ttft-ms where 'Requests without this header are placed behind all SLO requests, risking starvation.' Saturation detectors: utilization-detector (closed loop on real-time queue depth and KV cache, default, 'Highly accurate but subject to telemetry lag (thundering herd). In heterogeneous pools, it treats all endpoints equally (unweighted average), meaning a small saturated endpoint can trigger global backpressure') and concurrency-detector (open loop on in-flight accounting, 'Instantaneous reaction but blind to actual hardware memory pressure (KV cache filling)'). With flow control DISABLED (the default), saturation means 'sheddable requests (those with negative priority) are immediately rejected with HTTP 429 ... All other requests pass directly to the model servers.' The Healthy Buffer principle: keep a small queue ON the model servers 'just large enough to ensure continuous batching engines never starve for work, but small enough that the vast majority of queuing happens centrally in the EPP where priority and fairness can be enforced.' In-flight eviction targets only negative-priority in-flight requests and returns 429 with x-llm-d-request-dropped-reason: evicted; 'Flow Control assumes a bounded engine reclaim time and does not observe it directly, so verify on your engine that aborted requests free capacity promptly before enabling eviction in production', and the evicted caller owns retry so its path 'needs idempotency or request-level dedup so a retried request cannot produce two final results.' What It Does Not Solve, verbatim: 'Absolute Capacity Shortages: Flow Control only handles when and in what order requests are dispatched. It cannot make the pool faster or create capacity that does not exist.'; 'TTFT Shifts (Not Elimination) ... By enabling it, you make the explicit choice to protect TPOT at the expense of queue time.'; 'Non-Persistence: Queues are stored purely in-memory. If the EPP process restarts or fails, queued requests are lost. During a graceful shutdown, the EPP evicts queued requests with a retryable HTTP 503 ... whereas abrupt crashes will result in hard connection drops for the client.' Error mapping includes 503 rejected-shutting-down for graceful drain while the EPP is alive, 503 rejected-ttl-expired, 503 rejected-context-cancelled, 500 for a genuine internal error, with the IMPORTANT note 'These mappings only apply while the EPP is reachable.'