inference/research
verifiedhigh · 0.92updated 2026-10-04

Triton metrics doc - default endpoint :8002, the disable flags, and per-family type selection

repo

Fields

id
sources/w3m-triton-metrics-defaults-port
Publisher
NVIDIA Triton Inference Server
Kind
repo
Accessed
2026-10-04
notes
Rendered metrics.html from docs/user_guide/metrics.md in the main branch, read 2026-10-04. Verbatim: 'By default, these metrics are available at http://localhost:8002/metrics. The metrics are only available by accessing the endpoint, and are not pushed or published to any remote server.' Disable controls: 'The tritonserver --allow-metrics=false option can be used to disable all metric reporting, while the --allow-gpu-metrics=false and --allow-cpu-metrics=false can be used to disable just the GPU and CPU metrics respectively.' Latency families default to COUNTERS; the single histogram, nv_inference_first_response_histogram_ms, is NOT on by default and needs '--metrics-config histogram_latencies=true'. Summary variants need summary_latencies=true. Default histogram buckets are visible in the doc's own sample output: le=100, 500, 2000, 5000. Custom buckets are per-family via model_metrics[].metric_control[].metric_identifier.family in config.pbtxt, and the doc states nv_inference_first_response_histogram_ms is currently the only family supporting them. Also defines nv_inference_pending_request_count as 'incremented when a request is enqueued to the server (TRITONSERVER_ServerInferAsync) and decremented when a backend is about to start executing the request' - the honest queue-depth definition.

Notes

Rendered metrics.html from docs/user_guide/metrics.md in the main branch, read 2026-10-04. Verbatim: 'By default, these metrics are available at http://localhost:8002/metrics. The metrics are only available by accessing the endpoint, and are not pushed or published to any remote server.' Disable controls: 'The tritonserver --allow-metrics=false option can be used to disable all metric reporting, while the --allow-gpu-metrics=false and --allow-cpu-metrics=false can be used to disable just the GPU and CPU metrics respectively.' Latency families default to COUNTERS; the single histogram, nv_inference_first_response_histogram_ms, is NOT on by default and needs '--metrics-config histogram_latencies=true'. Summary variants need summary_latencies=true. Default histogram buckets are visible in the doc's own sample output: le=100, 500, 2000, 5000. Custom buckets are per-family via model_metrics[].metric_control[].metric_identifier.family in config.pbtxt, and the doc states nv_inference_first_response_histogram_ms is currently the only family supporting them. Also defines nv_inference_pending_request_count as 'incremented when a request is enqueued to the server (TRITONSERVER_ServerInferAsync) and decremented when a backend is about to start executing the request' - the honest queue-depth definition.

Cross-references

This record references no other records.