id
engines/niche-text-embeddings-inference
Repo
huggingface/text-embeddings-inference
Languages
RustPythonJavaScriptMakefileNix
Design goal
throughput-firstedge/mobilemulti-modelserverlessproduction
Scheduling
continuous-batchingtoken-based-dynamic-batchingprefix-cache
Notable features
A whole engine class of its own: embedding, reranking and sequence-classification serving. Nobody confuses this with a generative engine, which is exactly why HuggingFace built itRust core with a Python router; the README claims no model graph compilation step, small docker images and fast boot times, and sells 'true serverless' on the back of that - a real operational advantage over the PyTorch engines because cold-start is shortToken-based dynamic batching rather than request-count batching, which is the correct unit when inputs have wildly different token lengthsTwo weight-loading paths: safetensors AND onnx, plus ONNX Runtime for CPU. This is unusual and it is why TEI runs where a torch-based server will notOptimized transformer path using Flash Attention, Candle and cuBLASLt; local builds go through `cargo install --path router -F candle-cuda` (or -F candle-cuda-turing for Turing)Metal support for local execution on MacsProduction signals baked in: distributed tracing with OpenTelemetry, Prometheus metrics, gRPC alongside REST, and Swagger/OpenAPI docs at /docsCovers FlagEmbedding, Jina, GTE, E5, BGE-class models plus SPLADE sparse pooling and sequence classifiers for rerankingv1.9 line: v1.9.4 released 2026-09-15, last push 2026-09-29, 5069 stars, 457 open issues
Hardware caveats
THE HARDWARE MATRIX IS THE RECORD. Published docker image table, v1.9 line: CPU x86_64 and aarch64; Volta x86_64 explicitly NOT SUPPORTED; Turing (T4, RTX 2000) image exists but is labelled experimental; Ampere 8.0 and 8.6, Ada Lovelace, Hopper images are the supported CUDA targets; Blackwell 100 (B200/GB200), 120 (RTX 50-series) and 121 (DGX Spark GB10) images are all labelled experimentalThe README states plainly that GPUs with CUDA compute capability < 7.5 are not supported, naming V100, Titan V and the GTX 1000 series. If you have a V100 - still extremely common in inference fleets - TEI does not run and that is deliberate, not a bugNVIDIA drivers must be compatible with CUDA 12.2 or higherFlash Attention is turned OFF by default on the Turing image because of precision issues; you must set USE_FLASH_ATTENTION=True to get FA v1 back, and it is v1 not v2CUDA feature flags are per-architecture, so the source build requires you to know your compute capability before `cargo install` - you cannot build one binary for a mixed fleetROCm is a documented but narrower path: separate images for AMD Instinct MI200 and MI300 series only, with its own guide. No ROCm support is claimed for consumer AMD partsApple Silicon local use runs on CPU/Metal via the CPU build; the README's own wording is 'for ARM64 hosts without NVIDIA GPUs ... Inference runs on CPU cores'. Do not read 'Metal support' as Metal-accelerated GPU embedding servingArchitectures ahead of the supported list (Blackwell) run experimental images - i.e. unsupportedTEI serves embeddings and rerankers only. Point it at a generative model and it is the wrong tool; that is not a limitation so much as the boundary of the class
Best for
Serving embeddings and rerankers in production, especially where the vector DB is the bottleneck and embedding throughput is the real constraintServerless or autoscaling deployments, because no graph compilation and small images are first-class design goals rather than accidentsCPU-only or ARM64 embedding serving via the ONNX path, which is the cheapest credible way to embed at scale without a GPUHeterogeneous fleets where embedding models are small and the per-image CUDA specialisation is annoying - pick the right tag per SKUTeams already on HuggingFace infrastructure, since the model and pooling conventions match
Avoid for
V100 or older, and Blackwell-class parts on anything but experimental imagesGenerative serving of any kindAMD consumer GPUs (RX cards) - ROCm coverage is data-centre MI-series onlyExpecting the Mac path to be GPU-accelerated; it is a CPU build
notes
The embedding/reranking niche's reference runtime, and the answer to 'which engine do I use for retrieval infrastructure'. Rust core + ONNX/Candle/CUDA paths + token-based dynamic batching + explicit per-architecture image matrix makes it the best-documented hardware story in this slice. Related but distinct: [[niche-infinity]] (multi-backend, more model coverage, slower cadence) and [[niche-colbert]] (late interaction, a different compute shape entirely). All metadata from the GitHub API on 2026-10-03. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-23 (0.4 months before this record), latest release v1.9.4 published 2026-09-15, 68 contributors (distinct committers including anonymous), open_issues_count 213 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 5070 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. tracing_support=unknown (set 2026-10-04): same repo as hf-text-embeddings-inference (huggingface/text-embeddings-inference), whose record states distributed tracing with OpenTelemetry is present but does not establish span granularity. Deliberately not defaulted to none and deliberately not guessed upward. TRACING VERIFIED 2026-10-05, read at main 2026-10-05. router/src/lib.rs defines a ResponseMetadata struct holding start_time, tokenization_time, queue_time and inference_time, and BOTH surfaces consume it: record_span() writes all four onto the current tracing span, and record_metrics() feeds four separate Prometheus histograms (te_request_duration, te_request_tokenization_duration, te_request_queue_duration, te_request_inference_duration). Each HTTP handler in router/src/http/server.rs carries #[instrument(skip_all, fields(total_time, tokenization_time, queue_time, inference_time))] and takes Extension<Option<opentelemetry::Context>> then calls span.set_parent(context), so an inbound W3C traceparent is honoured. router/src/logging.rs installs the OTLP-over-gRPC pipeline; router/src/main.rs exposes --otlp-endpoint, --otlp-service-name (default text-embeddings-inference.server) and --disable-spans. Per-phase rather than per-request because tokenization, queueing and inference are separately attributable within one span - which is the distinction that lets a reader say whether a slow embedding request is the tokenizer, the admission semaphore or the model.