llm-d KEDA with EPP Metrics design (why utilisation fails, four demand signals, isolation, limitations)

Verdict on the naive signals: 'During active token generation, GPU utilization can remain high whether a model server is lightly loaded or saturated. A utilization-based autoscaler cannot reliably distinguish spare serving capacity from an overloaded batch.' Four signals, each with its own deployable guide: queue depth (llm_d_epp_flow_control_queue_size plus llm_d_epp_request_running, threshold is an absolute per-replica target), pool saturation (llm_d_epp_flow_control_pool_saturation plus llm_d_epp_request_running, 'A normalized ratio (0.0-1.0+), more portable across hardware'), token backlog (llm_d_epp_inflight_tokens plus per-pod KV cache occupancy, threshold is 'Seconds of prefill queue wait, derived from a share of the TTFT SLO and a calibrated peakPrefillThroughput'), estimated latency (EPP predicted/actual TTFT and TPOT histograms, 'Latency divided by SLO, with a hysteresis band'). Token-backlog rationale: 'request counts rate an 8192-token prompt the same as a 512-token one, while the prefill work differs by 16x.' Pipeline: EPP metrics scraped by Prometheus, evaluated by the KEDA Prometheus scaler into a scaler value, published through the Kubernetes External Metrics API, and consumed by an autoscaling/v2 HPA that KEDA creates and owns. 'The user creates a ScaledObject, not a second HPA. Multiple HPAs targeting the same Deployment conflict with one another.' Prometheus Adapter is not in this pipeline because KEDA queries Prometheus directly; 'The Prometheus Adapter path is deprecated.' Metric isolation: 'Each PromQL query aggregates to one value and must isolate the EPP/InferencePool associated with the target Deployment' and 'The running-request metric does not expose inference_pool.' Pending pods: a trigger can read kube_deployment_status_replicas and kube_deployment_status_replicas_ready, and the SLO-aware path applies an in-flight credit r/n because 'the latency signal is measured on ready pods only ... those pods over-report the post-warmup load while a scale-up is in flight.' Ownership: 'KEDA owns the generated HPA for the lifetime of the ScaledObject'; 'cooldownPeriod controls scale-down to zero; it does not control ordinary HPA scale-down while the minimum replica count is one or greater.' Contended accelerator budgets are handled BELOW the autoscaler by Kueue (per-model quota floors in a shared cohort, lending, preemption to reclaim a floor), with over-budget replicas staying Pending; the removed replica rebalancer 'must never run' with Kueue. Limitations: 'No cross-variant placement', 'No cost-aware choice of hardware', and 'Warmup is paid at every scale-up. Model load and torch-compile time sit between the decision and the capacity. Cutting pod-ready time is usually the highest-leverage tuning available.' Scale to zero: EPP Flow Control buffers requests while no endpoint is ready, a queued request raises the queue-size metric, KEDA activates the Deployment, and EPP dispatches after a model server becomes Ready; but 'Model cold-start time becomes request latency, and clients must allow enough time for the model to load.'