TGI batch decode-duration histogram (prefill/decode split)
text-generation-inference · histogram
Fields
id
metric-exposures/w3m-text-generation-inference-tgi-batch-decode-durationMetric name
tgi_batch_decode_duration
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:9000/metrics
Histogram buckets
0.00010.000150.0002250.00033750.000506250.000759380.001139060.001708590.002562890.003844340.00576650.008649760.012974630.019461950.029192930.043789390.065684080.098526130.147789190.221683780.332525670.498788510.748182761.122274151.683411222.525116833.787675245.681512878.522269312.7834039519.1751059228.7626588843.1439883364.7159824997.07397374
Labels
method
Enabled by default
yes
Phase scoped
yes
notes
TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. method in {prefill, decode}. All four batch duration families share the same suffix-matched ladder.
Notes
TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. method in {prefill, decode}. All four batch duration families share the same suffix-matched ladder.