Triton Inference Server: a token-throughput metric - VERIFIED ABSENT
triton-inference-server · unknown
Fields
id
metric-exposures/w3m-triton-inference-server-no-nv-inference-token-throughputMetric name
nv_inference_token_throughput
Metric type
unknown
Exposure
unknown
Verified negatively
yes
notes
VERIFIED NEGATIVE. Triton Inference Server does not emit a token-throughput metric. nv_inference_token_throughput is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. Triton counts REQUESTS and INFERENCES, not tokens: nv_inference_count counts a batch of n as n, nv_inference_exec_count counts batches. Token throughput is only available from a backend that registers token counters. What was searched: the docs' metric tables contain no token-count or tok/s family.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.
Notes
VERIFIED NEGATIVE. Triton Inference Server does not emit a token-throughput metric. nv_inference_token_throughput is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. Triton counts REQUESTS and INFERENCES, not tokens: nv_inference_count counts a batch of n as n, nv_inference_exec_count counts batches. Token throughput is only available from a backend that registers token counters. What was searched: the docs' metric tables contain no token-count or tok/s family.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.