id
engines/niche-triton-inference-server
Repo
triton-inference-server/server
Languages
PythonShellC++CMakeJavaGo
Design goal
productionmulti-modelmulti-frameworkthroughput-firstecosystem-lock-in
Scheduling
dynamic-batchingsequence-batchingimplicit-state-managementmodel-ensemblesconcurrent-model-execution
Backends
cudatensorrtonnxopenvinopythoncpuinferentia
Formats
onnxtensorflowpytorchplandalifil
Notable features
The general-purpose inference server, from before the LLM serving era, and still the default answer for 'I have many different models on many different frameworks'. vLLM and friends are LLM engines; Triton is the substrate they could sit onDynamic batching AND sequence batching with implicit state management for stateful models - the sequence batcher is the thing that made Triton relevant to LLM serving before paged attention existedConcurrent model execution: many models on one GPU with automatic concurrencyBackend API for custom backends, plus Python-based backends as the in-process extension point. This is how you get a new framework onto Triton without C++Model ensembles and Business Logic Scripting for multi-stage pipelines (preprocess, ensemble, postprocess) declared in the model repository rather than codedMulti-framework: PyTorch, TensorRT, ONNX Runtime, OpenVINO, Python, RAPIDS FIL and more per the READMERuns on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia, across cloud, data center, edge and embeddedHTTP/REST and gRPC protocols; filesystem model repository with explicit model-control modePart of NVIDIA AI Enterprise, which is the commercial context for supportBSD-3-Clause, which is a genuine contrast with the AGPL/NOASSERTION licences elsewhere in this slice11042 stars, v2.73.0 released 2026-09-30 (the 26.09 container on NGC), last push 2026-10-03, 4163 open issues
Hardware caveats
NOT AN LLM ENGINE. Triton has no paged KV, no prefix caching, no speculative decoding and no token-level scheduling. If you are serving LLMs at scale you want vLLM or SGLang; Triton is what you deploy when the answer is 'a directory of heterogeneous models'Vendor lock-in is the central design fact: the model repository format, the backends, the container releases and the support path are all NVIDIA's, and the README explicitly places Triton inside NVIDIA AI EnterpriseThe 26.09 container release and the r26.09 git branch are the versioning coupling to watch - README instructions clone branch r26.09, so Triton version lines track NVIDIA's calendar not semver cadencePython backends run in-process and are the pragmatic way to add a framework, but they are also the fastest way to hit GIL-bound throughput on a token loop. Fine for ensembles, bad for hot per-token pathsBackend maturity is uneven and version-specific. TensorRT plan files are locked to a specific TensorRT version; ONNX Runtime and OpenVINO backends track their own upstream. The 'supports many frameworks' claim is a list of independently-versioned dependenciesInferentia support is AWS-specific and inherently narrow4163 open issues; as a 2018-vintage project the accumulated surface is largeREADME feature claims are NVIDIA's own - the 'optimized performance' framing is vendor language and no per-model numbers are given
Best for
Serving a portfolio of many different models from many different frameworks behind one HTTP/gRPC contract - Triton's actual design targetNon-LLM workloads at scale: vision, speech, recommendation, classical ML alongside a couple of LLMsMulti-stage inference pipelines where preprocessing, the model and postprocessing belong in one declared ensembleDeploying to AWS Inferentia, or to edge/embedded NVIDIA hardwareTeams already licensed to NVIDIA AI Enterprise who want a supported substrate under their generative workloads
Avoid for
LLM serving at scale - no paged KV, no prefix caching, no speculative decoding. Use vLLM or SGLangToken-level throughput work where a Python backend's GIL is the bottleneckPortability beyond NVIDIA's ecosystem, unless you value the framework breadth enough to accept the model-repository formatTensorRT plan-file portability - plans are version-locked
Tracing support
per-request
notes
Included as the pre-LLM general-purpose server that the whole serving ecosystem grew around, and as the honest answer to 'why not just use Triton'. Its sequence batcher and implicit state management predate paged attention and are the historically interesting part. BSD-3-Clause, 2018 vintage, r26.09 branch/container coupling, and the AI Enterprise context are all recorded because they bear on adoption. All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-01 (0.1 months before this record), latest release v2.73.0 published 2026-09-30, 149 contributors (distinct committers including anonymous), open_issues_count 897 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 11044 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. tracing_support set to per-request on 2026-10-04: Triton exposes OpenTelemetry as an alternative trace backend (--trace-config mode=opentelemetry --trace-config opentelemetry,url=<endpoint>) with context propagation supported, and its documented minimal trace is 3 spans - top level, model and compute - carrying model_name, model_version, request_id and parent_id (obs-triton-trace-doc). Not per-phase: the three spans are nesting levels of one request, not prefill and decode, and Triton core has no phase-level tracing concept. Note the clock caveat carried from the same doc: OTel timestamps use the system clock while Triton's own trace API uses a steady clock, so the two modes are not numerically comparable.