inference/research
verifiedhigh · 0.94updated 2026-10-04

Triton+TRT-LLM TRT-LLM request gauge, label-switched

triton-inference-server · gauge

Fields

id
metric-exposures/w3m-triton-inference-server-nv-trt-llm-request-metrics
Metric name
nv_trt_llm_request_metrics
Metric type
gauge
Unit
1
Exposure
prometheus
Endpoint
:8002/metrics
Labels
modelversionrequest_type
Enabled by default
yes
notes
Triton main, docs/user_guide/metrics.md, read 2026-10-04. On by default: 'By default, these metrics are available at http://localhost:8002/metrics', and the endpoint is pure pull - 'not pushed or published to any remote server'. tritonserver --allow-metrics=false turns the whole thing off. Labels are model and version, and crucially the LATENCY families are per MODEL, per request-update - not per-request observations as in vLLM/SGLang/TGI. Backend-contributed via the Triton custom metrics API, NOT a Triton core metric. All six families share the model and version labels. Observed label values: ['waiting', 'context', 'scheduled', 'max', 'active']. One gauge carrying five different quantities as a LABEL. This is the structural difference from every engine-native set in this repo: you query it by filtering request_type, not by naming a metric, and the running/waiting distinction lives in request_type rather than in two metric names. Needs Triton 23.11+; before 23.12 the base nv_inference_* metrics report 0 for this backend.

Notes

Triton main, docs/user_guide/metrics.md, read 2026-10-04. On by default: 'By default, these metrics are available at http://localhost:8002/metrics', and the endpoint is pure pull - 'not pushed or published to any remote server'. tritonserver --allow-metrics=false turns the whole thing off. Labels are model and version, and crucially the LATENCY families are per MODEL, per request-update - not per-request observations as in vLLM/SGLang/TGI. Backend-contributed via the Triton custom metrics API, NOT a Triton core metric. All six families share the model and version labels. Observed label values: ['waiting', 'context', 'scheduled', 'max', 'active']. One gauge carrying five different quantities as a LABEL. This is the structural difference from every engine-native set in this repo: you query it by filtering request_type, not by naming a metric, and the running/waiting distinction lives in request_type rather than in two metric names. Needs Triton 23.11+; before 23.12 the base nv_inference_* metrics report 0 for this backend.

Cross-references