TGI batch forward-duration histogram (prefill/decode split)

Fields

id

metric-exposures/w3m-text-generation-inference-tgi-batch-forward-duration

Metric name

tgi_batch_forward_duration

Metric type

histogram

Unit

seconds

Exposure

prometheus

Endpoint

:9000/metrics

Histogram buckets

Labels

Enabled by default

yes

Phase scoped

yes

notes

TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. method in {prefill, decode}. This label is the reason TGI is worth knowing about: it is the only engine here that lets you compare prefill and decode batch cost directly as two series of one metric, rather than reconstructing the split from separate metric names.

Notes

TGI main, router/src/server.rs plus backends/v3/src/{backend,queue,lib}.rs, read 2026-10-04. On by default: the Prometheus recorder is built unconditionally with PrometheusBuilder and an HTTP listener on prometheus_port, default 9000, so nothing has to be opted into. Label keys are method (prefill|decode) on the batch families, err on tgi_request_failure, and reason on tgi_batch_concat - there are no model or engine labels, so every series is fleet-wide. method in {prefill, decode}. This label is the reason TGI is worth knowing about: it is the only engine here that lets you compare prefill and decode batch cost directly as two series of one metric, rather than reconstructing the split from separate metric names.

Cross-references

Engine (1)

Inference enginesText Generation Inference