Gateway API Inference Extension - Introduction and concepts (Kubernetes SIGs)

Fetched 2026-10-04. 'An official Kubernetes project that optimizes self-hosting Generative Models on Kubernetes.' Definitions verbatim: 'Inference Gateway: A proxy/load-balancer that has been coupled with the EndPointer Picker extension'; 'Inference Router: An extendable component that makes decisions about which endpoint is optimal (best cost / best performance) for an inference request based on Metrics and Capabilities from Model Serving'; 'Metrics and Capabilities: Data provided by model serving platforms about performance, availability and capabilities to optimize routing. Includes things like Prefix Cache status or LoRA Adapters availability.'; 'Endpoint Picker (EPP): An implementation of an Inference Router.' Mechanism verbatim: routing is done through Envoy's External Processing Protocol (ext_proc) so 'any gateway that supports both ext-proc and Gateway API' - named: Envoy Gateway, kgateway, GKE Gateway - becomes an inference gateway. Features verbatim: model-aware routing (by model name not just path, 'also extends to Low-Rank Adaptation (LoRA) fine-tuned models'), serving priority (latency-sensitive chat vs latency-tolerant summarization), model rollouts by traffic splitting on model names, and customizable load balancing using metrics emitted from model servers ('Model Server-aware load balancing ("smart" load balancing as it's sometimes referred to in this repo) has been proven to reduce the serving latency and improve utilization of accelerators'). Explicitly positions itself BELOW higher-level AI gateways: 'integrate your self-hosted models alongside model-as-a-service providers in a higher level AI Gateways like LiteLLM, Gloo AI Gateway, or Apigee.' That layering statement is the topology fact this repo needs. Fewest supporting: EPP is 'a lightweight EPP for conformance testing purposes.'