TGI per-request mean-time-per-token histogram
Fields
metric-exposures/w3m-text-generation-inference-tgi-request-mean-time-per-token-duNotes
TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. This is TGI's TPOT, and its denominator is the trap: the router computes time_per_token = inference_time / generated_tokens, so it INCLUDES prefill and is not comparable to vLLM's vllm:request_time_per_output_token_seconds or SGLang's sglang:request_time_per_output_token_seconds. Buckets are the shared 35-step geometric ladder because the matcher is Matcher::Suffix("duration") - in seconds, starting at 0.00015.