inference/research
verifiedhigh · 0.94updated 2026-10-04

Triton time-to-first-RESPONSE histogram

triton-inference-server · histogram

Fields

id
metric-exposures/w3m-triton-inference-server-nv-inference-first-response-histogra
Metric name
nv_inference_first_response_histogram_ms
Metric type
histogram
Unit
milliseconds
Exposure
prometheus
Endpoint
:8002/metrics
Histogram buckets
10050020005000
Labels
modelversion
Enabled by default
no
notes
Triton main, docs/user_guide/metrics.md, read 2026-10-04. On by default: 'By default, these metrics are available at http://localhost:8002/metrics', and the endpoint is pure pull - 'not pushed or published to any remote server'. tritonserver --allow-metrics=false turns the whole thing off. Labels are model and version, and crucially the LATENCY families are per MODEL, per request-update - not per-request observations as in vLLM/SGLang/TGI. OFF BY DEFAULT - requires --metrics-config. NOT ON BY DEFAULT: needs --metrics-config histogram_latencies=true. Default buckets are the four bounds visible in the docs' own sample output (100, 500, 2000, 5000 ms) and are overridable per family via model_metrics[].metric_control[] in config.pbtxt, with model reload required. This is NOT time to first token - see the verified-absence record for why that distinction matters.

Notes

Triton main, docs/user_guide/metrics.md, read 2026-10-04. On by default: 'By default, these metrics are available at http://localhost:8002/metrics', and the endpoint is pure pull - 'not pushed or published to any remote server'. tritonserver --allow-metrics=false turns the whole thing off. Labels are model and version, and crucially the LATENCY families are per MODEL, per request-update - not per-request observations as in vLLM/SGLang/TGI. OFF BY DEFAULT - requires --metrics-config. NOT ON BY DEFAULT: needs --metrics-config histogram_latencies=true. Default buckets are the four bounds visible in the docs' own sample output (100, 500, 2000, 5000 ms) and are overridable per family via model_metrics[].metric_control[] in config.pbtxt, with model reload required. This is NOT time to first token - see the verified-absence record for why that distinction matters.

Cross-references