inference/research
verifiedhigh · 0.93updated 2026-10-04

Triton Inference Server: a time-per-output-token / inter-token latency metric - VERIFIED ABSENT

triton-inference-server · unknown

Fields

id
metric-exposures/w3m-triton-inference-server-no-nv-inference-time-per-output-toke
Metric name
nv_inference_time_per_output_token
Metric type
unknown
Exposure
unknown
Verified negatively
yes
notes
VERIFIED NEGATIVE. Triton Inference Server does not emit a time-per-output-token / inter-token latency metric. nv_inference_time_per_output_token is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. No decode-rate metric of any kind exists in Triton core. Token-shaped latency has to come from the backend's custom metrics API, which for the TRT-LLM backend means a label-switched gauge family rather than a named metric. What was searched: the docs' metric tables contain no token-rate or inter-token family; the custom-metrics section points at the TensorRT-LLM and vLLM backends for LLM-specific metrics.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Notes

VERIFIED NEGATIVE. Triton Inference Server does not emit a time-per-output-token / inter-token latency metric. nv_inference_time_per_output_token is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. No decode-rate metric of any kind exists in Triton core. Token-shaped latency has to come from the backend's custom metrics API, which for the TRT-LLM backend means a label-switched gauge family rather than a named metric. What was searched: the docs' metric tables contain no token-rate or inter-token family; the custom-metrics section points at the TensorRT-LLM and vLLM backends for LLM-specific metrics.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Cross-references