inference/research
verifiedhigh · 0.94updated 2026-10-04

TensorRT-LLM request prefill-phase duration histogram

tensorrt-llm · histogram

Fields

id
metric-exposures/w3m-tensorrt-llm-trtllm-request-prefill-time-seconds
Metric name
trtllm_request_prefill_time_seconds
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:8000/prometheus/metrics
Histogram buckets
0.0010.0050.010.0250.050.10.250.512.55102040801606402560
Labels
model_nameengine_type
Enabled by default
no
Phase scoped
yes
notes
TensorRT-LLM main, MetricsCollector in tensorrt_llm/metrics/collector.py, read 2026-10-04. collector.py line 32 carries the header comment 'Adapted from https://github.com/vllm-project/vllm/blob/v0.10.0rc1/vllm/engine/metrics.py#L30', so this is a vLLM fork with the prefix swapped to trtllm_. NOT ON BY DEFAULT: the collector is only constructed and /prometheus/metrics only mounted when --return_perf-metrics is set (llm_args default False). Endpoint is :8000/prometheus/metrics - NOT /metrics, and the server's own GET /metrics serves a JSON iteration-stats snapshot, so a scraper pointed at /metrics gets JSON rather than exposition text. Labels are model_name and engine_type. The docstrings give the intervals explicitly, which is the useful part: prefill is (first_token_time - first_scheduled_time), decode is (last_token_time - first_token_time), inference is (last_token_time - first_scheduled_time). Prefill's ladder is finer at the low end (starts 0.001, steps 0.025) than the shared request ladder.

Notes

TensorRT-LLM main, MetricsCollector in tensorrt_llm/metrics/collector.py, read 2026-10-04. collector.py line 32 carries the header comment 'Adapted from https://github.com/vllm-project/vllm/blob/v0.10.0rc1/vllm/engine/metrics.py#L30', so this is a vLLM fork with the prefix swapped to trtllm_. NOT ON BY DEFAULT: the collector is only constructed and /prometheus/metrics only mounted when --return_perf-metrics is set (llm_args default False). Endpoint is :8000/prometheus/metrics - NOT /metrics, and the server's own GET /metrics serves a JSON iteration-stats snapshot, so a scraper pointed at /metrics gets JSON rather than exposition text. Labels are model_name and engine_type. The docstrings give the intervals explicitly, which is the useful part: prefill is (first_token_time - first_scheduled_time), decode is (last_token_time - first_token_time), inference is (last_token_time - first_scheduled_time). Prefill's ladder is finer at the low end (starts 0.001, steps 0.025) than the shared request ladder.

Cross-references