inference/research
verifiedhigh · 0.85updated 2026-10-04

Gateway API Inference Extension README (inference gateway, Endpoint Picker, stated roadmap gaps)

repo

Fields

id
sources/ops-gwie-inference-extension
Publisher
Kubernetes SIGs
Kind
repo
Published
main branch as of 2026-10-04
Accessed
2026-10-04
notes
Claims the inference gateway 'Improves the tail latency and throughput of LLM completion requests against Kubernetes-hosted model servers using an extensible request scheduling algorithm that is kv-cache and request cost aware, avoiding evictions or queueing as load increases'. The Roadmap is the honest part for an ops record: 'Fairness and priority between workloads within the same criticality band' and 'HPA support for autoscaling on aggregate metrics derived from the load balancer' are BOTH still listed as future work, as are 'Disaggregated serving support with independently scaling pools' and 'Heterogeneous accelerators'. So as of this commit the project does not claim cross-workload fairness or HPA-on-aggregates as shipped. Defines EPP as an ext-proc data-plane component that 'intercepts incoming inference requests and routes each request to the optimal model server replica' and describes an operational guardrail goal between different client model names. Note the API migration: the Endpoint Picker, InferenceObjective and InferenceModelRewrite APIs have moved to llm-d/llm-d-router and will be archived; this repo keeps only a lightweight EPP for conformance tests and the InferencePool API. The project is GA.

Notes

Claims the inference gateway 'Improves the tail latency and throughput of LLM completion requests against Kubernetes-hosted model servers using an extensible request scheduling algorithm that is kv-cache and request cost aware, avoiding evictions or queueing as load increases'. The Roadmap is the honest part for an ops record: 'Fairness and priority between workloads within the same criticality band' and 'HPA support for autoscaling on aggregate metrics derived from the load balancer' are BOTH still listed as future work, as are 'Disaggregated serving support with independently scaling pools' and 'Heterogeneous accelerators'. So as of this commit the project does not claim cross-workload fairness or HPA-on-aggregates as shipped. Defines EPP as an ext-proc data-plane component that 'intercepts incoming inference requests and routes each request to the optimal model server replica' and describes an operational guardrail goal between different client model names. Note the API migration: the Endpoint Picker, InferenceObjective and InferenceModelRewrite APIs have moved to llm-d/llm-d-router and will be archived; this repo keeps only a lightweight EPP for conformance tests and the InferencePool API. The project is GA.

Cross-references

This record references no other records.