inference/research
verifiedhigh · 0.92updated 2026-10-04

TensorRT-LLM: an inter-token-latency histogram - VERIFIED ABSENT

tensorrt-llm · unknown

Fields

id
metric-exposures/w3m-tensorrt-llm-no-trtllm-inter-token-latency-seconds
Metric name
trtllm_inter_token_latency_seconds
Metric type
unknown
Exposure
unknown
Verified negatively
yes
notes
VERIFIED NEGATIVE. TensorRT-LLM does not emit an inter-token-latency histogram. trtllm_inter_token_latency_seconds is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. TRT-LLM has TPOT (trtllm_time_per_output_token_seconds) but not the per-step ITL histogram vLLM and SGLang both have. The gap is real but narrow: TPOT covers per-request decode rate, and what is missing is the per-step gap that reveals a single stalled iteration averaged away across a request. What was searched: case-insensitive count of 'inter_token' and 'iter_token' in collector.py returns 0; the collector's docstring enumeration lists no ITL metric.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Notes

VERIFIED NEGATIVE. TensorRT-LLM does not emit an inter-token-latency histogram. trtllm_inter_token_latency_seconds is NOT a metric this engine registers - it is recorded here under the conventional name a reader would go looking for so the absence is queryable rather than implied by a gap in a list. TRT-LLM has TPOT (trtllm_time_per_output_token_seconds) but not the per-step ITL histogram vLLM and SGLang both have. The gap is real but narrow: TPOT covers per-request decode rate, and what is missing is the per-step gap that reveals a single stalled iteration averaged away across a request. What was searched: case-insensitive count of 'inter_token' and 'iter_token' in collector.py returns 0; the collector's docstring enumeration lists no ITL metric.. Do not read this record's metric_name as something the engine exports; it is the name that is missing.

Cross-references