Design goal
productionmulti-modelkubernetes-nativestandardized-inference
Scheduling
request-routingprefix-cachedisaggregated-prefill-decodeKV-aware-routing
Backends
none (orchestration layer - drives vLLM, SGLang, TensorRT-LLM, Triton)
Formats
passthrough (does not load weights - orchestrates inference engines)
Notable features
CNCF incubating project (joined 2025-11-11) - the Kubernetes-native serving standardLLMInferenceService CRD with prefix-aware routing, disaggregated serving, and fine-grained GPU schedulingInferencePool and InferenceModel CRDs for multi-model routing with EPP (Endpoint Picker Pod)Direct KEDA scaling support (v0.21.0) with idleReplicaCount=0 for true scale-to-zeroCanary traffic splitting for inference servicesNixlConnector for P/D engines so KV actually transfersDRA (Dynamic Resource Allocation) support via resourceClaimsOCI model delivery with measured startup benchmarks6070 stars, active development with 100+ PRs per release
Hardware caveats
Kubernetes is not optional - the entire architecture is built around K8s CRDs and controllersLLMInferenceService is v1alpha2 - still alpha API surfaceRequires Envoy Gateway for traffic routing - additional infrastructure dependencyEPP (Endpoint Picker Pod) adds a hop to every request - latency cost for routing intelligenceKEDA integration is new (v0.21.0) and not yet battle-tested in productionMulti-node disaggregated serving requires RDMA-capable fabric
Best for
Kubernetes-native LLM serving with standardized APIsMulti-model fleets where requests should route to the model with the best cache hitTeams already invested in Kubernetes who want a vendor-neutral serving layerAutoscaling on LLM-specific metrics (KV cache utilization, queue depth) via KEDA
Avoid for
Non-Kubernetes environments - the entire architecture assumes K8sSingle-model single-node serving where the overhead is not justifiedTeams needing a stable GA API - LLMInferenceService is still alphaLow-latency single-request serving where the EPP hop matters
notes
KServe is the Kubernetes-native inference orchestration layer. It does not run inference itself - it drives vLLM, SGLang, TensorRT-LLM, and Triton Inference Server. The LLMInferenceService CRD is the key abstraction for LLM-specific features: prefix-aware routing (send requests to the replica with the best KV cache hit), disaggregated serving (separate prefill/decode pools), and fine-grained GPU scheduling. The EPP (Endpoint Picker Pod) is the routing brain - it makes per-request routing decisions based on cache state, load, and latency. KServe v0.21.0 (2026-09-25) added direct KEDA scaling with idleReplicaCount=0 for true scale-to-zero, which is the first production-ready path for LLM-specific autoscaling on Kubernetes. The project joined CNCF as an incubating project in November 2025. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-03 (0.0 months before this record), latest release v0.21.0 published 2026-09-25, 389 contributors (distinct committers including anonymous), open_issues_count 214 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 6074 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set.