Request-count rate limiting is the wrong unit for LLM serving: one 8192-token prompt is 16x the prefill work of a 512-token one, and a request counter rates them identically

A per-tenant RPS quota that looks correctly sized lets one tenant saturate the fleet with a modest request rate of very long prompts, while a tenant sending many short requests never approaches its quota despite consuming comparable compute. Autoscaling driven by request counts under-provisions for the long-prompt tenant and over-provisions for the short-prompt one; the same RPS produces 'a third of a replica or three replicas of prefill work'. A fairness complaint from a small tenant arrives before any quota is technically exceeded.

llm-d states the argument exactly: 'Traditional API gateways rate-limit by Requests Per Second (RPS), assuming every request consumes roughly the same resources. In LLM serving, this approach is ill-suited. Resource consumption is driven by long input contexts and the unpredictable autoregressive decode loop (output tokens).' Its named failure modes are Resource Asymmetry ('A single request with a massive context can consume more resources than hundreds of short requests') and The Noisy Neighbor. The token-aware scaling guide quantifies the miscount: 'An 8192-token prompt is 16x the prefill work of a 512-token one, and a request counter rates them the same', giving the 16x figure from 8192/512. The correct prefill signal is uncached prompt tokens divided by a measured prefill token velocity, with 'Uncached' mattering because 'a prefix already in the KV cache is not prefilled again, so prefix caching lowers the effective ISL without changing the request'. Decode is a different unit entirely — occupancy, compared directly against kv cache utilisation, since it has no equivalent measured rate.

Quota and autoscaling in tokens, not requests, on both phases: prefill on queued uncached tokens divided by a calibrated peakPrefillThroughput, decode on vllm:kv_cache_usage_perc. If you cannot calibrate, count output-token budget as well as prompt tokens, and note that output length is the autoregressive half of the unpredictability. This is a partial answer, not a complete one: a token-rate quota bounds work but says nothing about a single request's peak, which is why sgl-router's admission design explicitly excludes request size from admission (see ops-gateway-admission-has-no-reservation).

Genuinely unsolved: a token-based rate limit is a rate, and a single request's peak footprint is unbounded within it. The llm-d per-band maxBytes and maxRequests give memory isolation between priority bands but are queue-capacity limits, not per-tenant compute quotas, and the fair-share metrics that exist (llm_d_epp_program_aware_jains_fairness_index over average wait time, program_aware_avg_wait_time_milliseconds) are observability, not enforcement of a token budget. The Gateway API Inference Extension roadmap still lists 'Fairness and priority between workloads within the same criticality band' as future work, so as of this commit the upstream project does not claim shipped cross-workload fairness. SCHEME RECLASSIFIED (2026-10-04): was 'config'.