Triton Inference Server

COMPUTE-BOUND for the encoder phase at batch 1, the same structural fact as every other encoder here: a BERT-size encoder does one bidirectional attention pass over all input tokens with no KV cache and no autoregressive streaming, so its arithmetic is dense GEMM with high arithmetic intensity and it wants FLOPS rather than HBM bandwidth. This is the exact inverse of an LLM decode part on the same GPU, where one weight is streamed per generated token at roughly 1 flop per byte and bandwidth alone decides tok/s. The practical hardware-selection consequence for a Triton deployment: the right box for an embedding/rerank tier is a high-FLOPS, high-core-count part (or simply many cheap cores), and buying HBM bandwidth for it is buying the wrong resource, whereas that same GPU bought for a decode tier would be justified by exactly that bandwidth. Two Triton-specific nuances are worth recording. First, cross-encoder rerankers take a correlated (query, passage) pair, so Triton reaches for its sequence batcher and implicit state management rather than plain dynamic batching, which means rerank serving is configured differently from embedding serving in the same server and the batching knobs do not transfer between the two phases. Second, the ensemble/BLS capability is where retrieval serving genuinely diverges from LLM serving: the vector-database lookup is not a model at all, it is a python business-logic step, which is exactly why a retrieval pipeline needs a general serving layer rather than a purpose-built encoder server. Can serve: anything expressible as an ONNX/PyTorch/TensorRT/OpenVINO/FIL graph, so all dense bi-encoders (BERT, XLM-RoBERTa, BGE, E5, GTE), cross-encoder rerankers (bge-reranker class, jina-reranker, the cross-encoder/ms-marco family), SPLADE sparse encoders via ONNX, CLIP-family multimodal encoders, and even a late-interaction ColBERT encoder if you accept token-wise output as a custom output tensor. Cannot serve: token-by-token generative decoding without the TensorRT-LLM backend or a custom Python backend; and Triton provides no pooling abstraction, so CLS versus MEAN versus LAST is your config problem rather than a framework feature, which is a real ergonomic difference from TEI. Confidence 0.8: the backend list, batching features, protocol and release pinning are quoted from the README, but Triton is a container platform rather than a single inference implementation, so what it can serve is bounded by its backends rather than by any model list. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-01 (0.1 months), latest release v2.73.0 published 2026-09-30, 149 contributors, open_issues_count 897 (issues PLUS pull requests), NOT archived, 11044 stars. Actively developed under NVIDIA stewardship. DEPLOYMENT: graded as INFRASTRUCTURE-IMPLIED rather than evidenced - this engine's entire purpose is production serving and it is embedded in NVIDIA's Triton Inference Server product line, but NVIDIA publishes no request volume, QPS or fleet size for it, and no independent third-party benchmark of the engine as deployed was found. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]]. FEATURE AXES: all seven are UNSTATED for this engine as of 2026-10-04 - no speculative decoding, chunked prefill, disaggregated P/D or structured output claim was confirmed from the README. That is the largest remaining axis gap on an engine a serious reader will evaluate, and it is the first gap worth closing. Its existing 'dynamic-batching, sequence-batching, ensemble-pipelines' scheduling entry is a real scheduling capability and is NOT the same as chunked prefill. tracing_support set to per-request on 2026-10-04: Triton exposes OpenTelemetry as an alternative trace backend (--trace-config mode=opentelemetry --trace-config opentelemetry,url=<endpoint>) with context propagation supported, and its documented minimal trace is 3 spans - top level, model and compute - carrying model_name, model_version, request_id and parent_id (obs-triton-trace-doc). Not per-phase: the three spans are nesting levels of one request, not prefill and decode, and Triton core has no phase-level tracing concept. Note the clock caveat carried from the same doc: OTel timestamps use the system clock while Triton's own trace API uses a steady clock, so the two modes are not numerically comparable.