inference/research
verifiedhigh · 0.94updated 2026-10-04

TGI per-request speculated-token histogram

text-generation-inference · histogram

Fields

id
metric-exposures/w3m-text-generation-inference-tgi-request-skipped-tokens
Metric name
tgi_request_skipped_tokens
Metric type
histogram
Unit
tokens
Exposure
prometheus
Endpoint
:9000/metrics
Histogram buckets
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100
Enabled by default
yes
notes
TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. tgi_request_input_length is matched by Matcher::Full with 100 linear buckets up to max_input_tokens; generated_tokens and max_new_tokens likewise up to max_total_tokens, so these three ladders depend on deployment sizing rather than being fixed.

Notes

TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. tgi_request_input_length is matched by Matcher::Full with 100 linear buckets up to max_input_tokens; generated_tokens and max_new_tokens likewise up to max_total_tokens, so these three ladders depend on deployment sizing rather than being fixed.

Cross-references