id
engines/niche-vllm-production-stack
Repo
vllm-project/production-stack
Languages
PythonShellGoHCLJinja
Design goal
productionmulti-modelease-of-usereference-deployment
Scheduling
prefix-cachekv-cache-offloadrequest-routing
Notable features
The reference answer to 'I have vLLM on Kubernetes and now what'. First released 2025-01-21; it is explicitly a reference implementation rather than a productIts own three goals, quoted from the README: scale from a single vLLM instance to a distributed deployment without changing application code; monitor through a web dashboard; and get the performance benefit of request routing plus KV-cache offloadingRequest routing and KV-cache offloading via LMCache are the substantive parts - this is the layer that turns N vLLM pods into one service, and prefix-cache reuse across pods is the reason it existsHelm-based install with a documented tutorial sequence from zero: install Kubernetes tooling, deploy on AWS/GCP/Lambda Labs/Azure, minimal Helm installation, customising vLLM configs, loading weights from a PersistentVolume, launching multiple different LLMs, then enabling KV cache offloading with LMCacheCommunity-driven with an official docs site (docs.vllm.ai/projects/production-stack), its own Slack channel and a published 2026 roadmapOfficial documentation released alongside the project; cloud deployment tutorials for three providers2654 stars, release vllm-stack-0.1.13 on 2026-09-29, last push 2026-10-02, 322 open issues
Hardware caveats
STILL ON A 0.1.x VERSION LINE. Latest release is vllm-stack-0.1.13 (2026-09-29) roughly twenty months after the initial 2025-01-21 release. That is the maturity signal and it matters: reference deployments get adopted by copying, and a 0.1.x line implies breaking changes between minorsKubernetes is not optional. The tutorial sequence starts at 'install Kubernetes (kubectl, helm, minikube)' and the whole thing is Helm charts plus HCL. This is not a path for a single box or a small team without a clustervLLM-only by construction. It is a deployment stack for vLLM specifically, so adopting it is adopting vLLM's hardware support and its release cadence as well - including whatever backend plugins that version supports (see [[niche-vllm-ascend]] for the pattern)Routing and KV-offload behaviour is not free: prefix-cache reuse across pods depends on LMCache and a shared store, and a misconfigured cache store shows up as latency, not as an errorThe performance benefit the README advertises is measured with the project's own configuration. Copying the Helm defaults is not the same as tuning routing for your trafficCloud tutorials cover AWS, GCP and Lambda Labs specifically; other providers are untested pathsHCL and Go alongside Python means the config surface is wider than a single-language project, which is the usual cost of a Helm-first deployment tool
Best for
Deploying vLLM to Kubernetes in a way that survives contact with a real traffic pattern - this is the maintained reference and it beats hand-rolled manifestsMulti-model serving where requests should route to the model that already has the prompt cachedGetting LMCache-based KV offloading working before designing your own cross-pod cacheFollowing the tutorial sequence end to end when you have never run vLLM in a cluster
Avoid for
Anything without Kubernetes - the toolchain is Helm-first and not optionalNon-vLLM engines; there is no abstraction over serving engines hereProduction deployments that need a stable semver line, given 0.1.x after twenty monthsCloud providers other than AWS/GCP/Lambda Labs without expecting to debug the tutorial yourself
notes
One of four infrastructure-layer records here (with [[niche-triton-inference-server]], [[niche-deepspeed-mii]] and, indirectly, [[niche-kserve]]). Included because the gap it fills is real and named: the difference between 'vLLM works' and 'vLLM works as a service', which is request routing plus cross-pod prefix-cache reuse. vllm-stack-0.1.13 recorded as the maturity caveat. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-02 (0.1 months before this record), latest release vllm-stack-0.1.13 published 2026-09-29, 163 contributors (distinct committers including anonymous), open_issues_count 233 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 2657 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ROUTER DETAIL ADDED 2026-10-04 from the in-tree vllm_router README (dep-vllm-prodstack-router-readme), which was not previously captured in this record. The router is a separate FastAPI/uvicorn process with a service-discovery and health-check surface of its own: --service-discovery {static|k8s}, --static-backends, --static-models, --static-aliases, --static-backend-health-checks (sends periodic dummy requests to verify models work), --k8s-port (default 8000), --k8s-namespace, --k8s-label-selector, --routing-logic {roundrobin|session}, --session-key, --engine-stats-interval (default 30 s scrape of per-engine metrics: QPS, TTFT, pending/running/finished requests, uptime), --request-stats-window (default 60 s sliding window), and hot-reloadable --dynamic-config-yaml/json. Two maturity signals worth carrying: the router README still lists prefix-aware routing as '(WIP)' while the docs site ships a prefix-aware tutorial and the code has a PrefixAwareRouter class, so the README lags the implementation; and prefix-aware routing is documented as routing by shared PROMPT PREFIX to maximise KV cache reuse - the tutorial's own demonstration is two requests with a common prefix where the second returns an LMCache hit on the first's blocks. Prefix-cache placement is where this layer's real failure modes live, and three are recorded as gotchas in this repo: dep-router-metric-name-parse-miss-looks-healthy, dep-prefixaware-subthreshold-latches-a-bad-first-pick and dep-router-prefix-tree-ram-is-the-router-oom.