HuggingFace Text Embeddings Inference (TEI)

COMPUTE-BOUND, NOT MEMORY-BOUND, at batch 1 for a BERT-size encoder - and this inverts the hardware-selection rule that applies to an LLM decode part. Autoregressive decode at batch 1 is memory-bound because exactly one weight is streamed per generated token and the arithmetic intensity is ~1 flop per byte, so HBM bandwidth decides tok/s. An encoder has no autoregressive step: every input token gets a full bidirectional attention pass in one shot, with no KV cache to reuse and no weight streaming to amortize. The dominant costs are the same GEMMs an LLM prefill does (attention QK^T and AV are O(L^2) with no causal halving, since both halves are computed) plus a pooling reduction that is memory-cheap. So the encoder wants FLOPS - dense low-precision tensor throughput - not bandwidth. Practical consequences recorded here: (a) the README's own guidance to raise --max-batch-tokens 'until the model is compute bound' is an explicit statement that batching is how you convert spare bandwidth into throughput, and that beyond that point more batch does nothing; (b) arithmetic-intensity crossover for a bge-base-class model at 512 tokens sits around batch 8-32, far lower than for LLM decode, which is why token-based dynamic batching is the whole game here; (c) a bandwidth-optimized part (large HBM, modest FLOPS) is the wrong buy for a dedicated embedding box, and a high-FLOPS low-capacity part can be the right one because a 110M encoder in fp16 is ~220 MB of weights that fit in almost any cache hierarchy. This is also why TEI's insistence on no graph compilation matters less for embeddings than for generation: the arithmetic is dense GEMM, which tensor cores handle generically, so a JIT-free Rust path costs little. Reranking sits even more firmly on the compute side than embedding: a cross-encoder must run the FULL query+passage pair through every layer, so its FLOP count scales with (query_len + passage_len) and there is no early exit and no cache to amortize - bge-reranker-large on a top-100 candidate set is roughly 100x the FLOPs of embedding those 100 passages once. That is why reranking is normally restricted to a short candidate list while embedding runs over the whole corpus. TeiBackend variants: the README documents the backend selection indirectly through cargo features and Docker image variants rather than a single named enum - candle (default Rust), ort (ONNX Runtime), mkl (Intel CPU), metal (Apple), and candle-cuda / candle-cuda-turing. 'candle' is recorded in supported_backends implicitly as the CPU-default path; ort and mkl are listed because the README names them as the recommended x86 CPU builds. supported_formats lists onnx because the README states ONNX weight loading; safetensors likewise. Confidence is 0.88 rather than higher because the route surface and the pooling/dtype options come from docs/openapi.json and --help output rather than prose, and because TEI's model support moves faster than its docs. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-09-23 (0.4 months before this record), latest release v1.9.4 published 2026-09-15, 68 contributors (distinct committers including anonymous), open_issues_count 213 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 5070 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. tracing_support=unknown (set 2026-10-04), deliberately NOT defaulted to none: the record's own feature text establishes that OpenTelemetry tracing exists here (--otlp-endpoint) alongside a Prometheus /metrics endpoint, but the SPAN GRANULARITY - one span per request, several, or per-phase - was not established from a source, and this enum encodes granularity. Raising it needs the router's tracing setup read, not inferred. TRACING VERIFIED 2026-10-05, read at main 2026-10-05. router/src/lib.rs defines a ResponseMetadata struct holding start_time, tokenization_time, queue_time and inference_time, and BOTH surfaces consume it: record_span() writes all four onto the current tracing span, and record_metrics() feeds four separate Prometheus histograms (te_request_duration, te_request_tokenization_duration, te_request_queue_duration, te_request_inference_duration). Each HTTP handler in router/src/http/server.rs carries #[instrument(skip_all, fields(total_time, tokenization_time, queue_time, inference_time))] and takes Extension<Option<opentelemetry::Context>> then calls span.set_parent(context), so an inbound W3C traceparent is honoured. router/src/logging.rs installs the OTLP-over-gRPC pipeline; router/src/main.rs exposes --otlp-endpoint, --otlp-service-name (default text-embeddings-inference.server) and --disable-spans. Per-phase rather than per-request because tokenization, queueing and inference are separately attributable within one span - which is the distinction that lets a reader say whether a slow embedding request is the tokenizer, the admission semaphore or the model.