One tenant's long context evicts every other tenant's KV cache: a single request can consume more KV than hundreds of short ones, so a quota on request count does not protect anyone

A small number of users sending long-context requests degrades the service for everyone: their p50 latency is fine while other tenants' p50 and p99 climb sharply, and the other tenants' requests are the ones being preempted and re-prefilled (see ops-kv-cache-full-preempts-and-recomputes). No tenant has exceeded their quota. Fair-share metrics show one program with enormous average wait time and the rest collapsing. The escalation is always 'their request rate is fine', and it is: their request rate is not the problem.

Capacity is consumed per token, not per request, and the engines' own victim selection is length-biased against long-prompt requests. llm-d names both halves: 'A single tenant sending massive prompts can monopolize an endpoint's KV cache and queue, starving others' and 'A single request with a massive context can consume more resources than hundreds of short requests.' SGLang's default --retraction-policy makes it explicit — 'length preserves the existing behavior and retracts short-output, long-input requests first' — so under KV pressure the engine preferentially sacrifices exactly the requests long-context tenants are sending. Compounding this, prefix caching makes cache residency a function of traffic history: a tenant with a hot shared prefix keeps a large slice of the cache permanently occupied, so its cost is invisible in both the request rate and the KV utilisation gauge of the moment.

Isolate at the routing layer with per-tenant fairness rather than hoping the engine shares fairly: llm-d Flow Control keys every request on a FlowKey of (FairnessID, Priority) with FairnessID extracted from the x-llm-d-inference-fairness-id header, and the round-robin fairness policy 'Guarantees fair sharing by cycling through active flows one by one ... Prevent[s] a single high-volume flow from starving others (solving the "Noisy Neighbor" problem)'. Add per-band memory isolation so a lower-priority tenant cannot consume the queue reserved for higher-priority ones — the global-strict policy deliberately does the opposite, 'Ignores flow isolation and serves all requests in a single global order'. Inside the engine, bound the worst case: cap max_model_len per tenant class, use max_num_seqs, and enable priority scheduling with a non-default retraction policy so eviction order is by priority rather than by length. Monitor fairness explicitly: llm_d_epp_program_aware_jains_fairness_index measures Jain's fairness over average wait time across active programs, with program_aware_avg_wait_time_milliseconds per program. Add cost to the quota itself (ops-ratelimit-token-vs-request-count).

Upstream does not claim to have solved this: the Gateway API Inference Extension roadmap still lists 'Fairness and priority between workloads within the same criticality band' as future work. And Flow Control is behind a feature gate, so the mitigation is opt-in infrastructure, not something a stock llm-d install provides. One further honest gap: Flow Control 'cannot remove wait time when the system is over capacity' — it redistributes it between tenants, so in a genuinely undersized pool fairness makes everyone slow instead of making anyone fast. SCHEME RECLASSIFIED (2026-10-04): was 'config'.