inference/research
verifiedhigh · 0.90updated 2026-10-04

TensorRT-LLM (observability surface: Prometheus metrics + OpenTelemetry)

cuda

Fields

id
engines/obs-tensorrt-llm-observability
Repo
NVIDIA/TensorRT-LLM
Languages
PythonC++CUDA
License
Apache-2.0
First release
2022
Design goal
observabilitynvidia-throughputdisaggregated-serving
Scheduling
inflight-batchingpaged-kvin-flight
Backends
cuda
Formats
safetensors
Notable features
The metric set is an acknowledged fork of vLLM's - collector.py carries the header comment 'Adapted from https://github.com/vllm-project/vllm/blob/v0.10.0rc1/vllm/engine/metrics.py#L30' - so the metric names are the vLLM names with the prefix swapped to `trtllm_`All the latency metrics a serving SLO needs exist: `trtllm_time_to_first_token_seconds`, `trtllm_time_per_output_token_seconds`, `trtllm_request_queue_time_seconds`, `trtllm_e2e_request_latency_seconds`, `trtllm_request_prefill_time_seconds`, `trtllm_request_decode_time_seconds`, `trtllm_request_inference_time_seconds`Infight-batching internals exposed as gauges: `trtllm_num_context_requests`, `trtllm_num_generation_requests`, `trtllm_num_scheduled_requests`, `trtllm_num_paused_requests`, `trtllm_total_context_tokens`, `trtllm_avg_decoded_tokens_per_iter`, `trtllm_iteration_latency_seconds`KV cache as both a ratio and raw block counts: `trtllm_kv_cache_utilization` (usedNumBlocks / maxNumBlocks), `trtllm_kv_cache_max_blocks`, `trtllm_kv_cache_free_blocks`, `trtllm_kv_cache_used_blocks`, `trtllm_kv_cache_tokens_per_block`, plus multi-tier `trtllm_kv_cache_host_utilization` and disk-prefetch counters for the KV offload pathPrompt-cache accounting is unusually careful about what the denominator is: `trtllm_prompt_cached_tokens_total` is documented as 'Process-lifetime total prompt tokens served from KV cache. Resets only when the serving process restarts, not when the KV cache is flushed or evicted'OpenTelemetry via `trtllm-serve --otlp_traces_endpoint`, with the guide also covering proxy configuration for disaggregated serving
Hardware caveats
All seven latency histograms take their bucket boundaries as constructor arguments (e2e_request_latency_buckets, time_to_first_token_buckets, time_per_output_token_buckets, request_queue_time_buckets, request_prefill_time_buckets, request_decode_time_buckets, request_inference_time_buckets). The library ships no defaults - the caller must supply them, and they are validated as strictly increasing and non-empty. A deployment that passes coarse buckets cannot resolve a tight TTFT SLOThe OTel packages are NOT bundled: the guide tells you to pip install opentelemetry-sdk, -api, -exporter-otlp and -semantic-conventions-ai yourself. Contrast vLLM, which bundles themNO preemption metric. grep for 'preempt' in the collector returns nothing, and the only related gauge is `trtllm_num_paused_requests` fed from the ifb_stats 'numPausedRequests' field - that is disaggregation pausing, not scheduler preemption. If you run TensorRT-LLM under KV pressure you cannot see it in metricsPer-position speculative acceptance counters are capped at MAX_SPEC_DECODE_POSITION_LABELS = 16 so label cardinality stays bounded; deep drafts are truncated at the metrics layer even though the per-request arrays themselves growGPU memory is reported in raw bytes (`trtllm_gpu_memory_usage_bytes`), so a percentage alert has to be computed by the reader against the card's capacity
Best for
An NVIDIA-only fleet where you want vLLM-compatible metric semantics with TensorRT-LLM's extra inflight-batching and multi-tier KV countersMeasuring multi-tier KV offload (GPU/host/disk), where `trtllm_kv_cache_host_utilization`, `trtllm_kv_cache_disk_prefetch_blocks_total` and `trtllm_kv_cache_onboard_bytes_total` have no counterpart in vLLMTriton deployments, where the same engine metrics are re-registered into the Triton endpoint under the `nv_trt_llm_` prefix
Avoid for
Observing preemption - there is no metric for it at allAny prebuilt histogram expectations: you have to pass buckets yourselfAssuming vLLM's bucket boundaries carry over - the names match, the buckets do not
Tracing support
per-request
Docs
https://nvidia.github.io/TensorRT-LLM/performance/metrics.html
notes
Metric list transcribed from the MetricsCollector class docstring and the constructor calls in tensorrt_llm/metrics/collector.py on main. Per-request: `trtllm_request_success_total` (label finished_reason), `trtllm_e2e_request_latency_seconds`, `trtllm_time_to_first_token_seconds`, `trtllm_time_per_output_token_seconds`, `trtllm_request_queue_time_seconds`, `trtllm_request_prefill_time_seconds`, `trtllm_request_decode_time_seconds`, `trtllm_request_inference_time_seconds`, `trtllm_prompt_tokens_total`, `trtllm_generation_tokens_total`, `trtllm_prompt_cached_tokens_total`, `trtllm_prompt_cache_hit_tokens_total` (labels cache_level, cache_tier), `trtllm_prompt_cached_tokens_per_request`, `trtllm_request_error_total`, `trtllm_prefill_perplexity`, `trtllm_generation_perplexity`. Iteration/scheduler: `trtllm_kv_cache_hit_rate`, `trtllm_kv_cache_utilization`, `trtllm_kv_cache_host_utilization`, `trtllm_kv_cache_iter_reuse_rate`, `trtllm_kv_cache_reused_blocks`, `trtllm_kv_cache_missed_blocks`, `trtllm_kv_cache_iter_reused_blocks`, `trtllm_kv_cache_iter_full_reused_blocks`, `trtllm_kv_cache_iter_partial_reused_blocks`, `trtllm_kv_cache_iter_missed_blocks`, `trtllm_kv_cache_gen_alloc_blocks_total`, `trtllm_kv_cache_onboard_bytes_total`, `trtllm_kv_cache_offload_bytes_total`, `trtllm_kv_cache_disk_prefetch_blocks_total`, `trtllm_kv_cache_intra_device_copy_bytes_total`, `trtllm_num_requests_running`, `trtllm_num_requests_waiting`, `trtllm_num_requests_completed_total`, `trtllm_max_num_active_requests`, `trtllm_iteration_latency_seconds`, `trtllm_gpu_memory_usage_bytes`, `trtllm_cpu_memory_usage_bytes`, `trtllm_pinned_memory_usage_bytes`, `trtllm_max_batch_size_static`, `trtllm_max_batch_size_runtime`, `trtllm_max_num_tokens_runtime`, `trtllm_kv_cache_max_blocks`, `trtllm_kv_cache_free_blocks`, `trtllm_kv_cache_used_blocks`, `trtllm_kv_cache_tokens_per_block`, `trtllm_num_context_requests`, `trtllm_num_generation_requests`, `trtllm_num_paused_requests`, `trtllm_num_scheduled_requests`, `trtllm_total_context_tokens`, `trtllm_avg_decoded_tokens_per_iter`. Spec decode: `trtllm_spec_decode_num_draft_tokens_total`, `trtllm_spec_decode_num_accepted_tokens_total`, `trtllm_spec_decode_acceptance_length`, `trtllm_spec_decode_draft_overhead`. Config-info gauges logged once at startup: `trtllm_model_config_info`, `trtllm_parallel_config_info`, `trtllm_speculative_config_info`, `trtllm_kv_cache_config_info`. Note the docstring lists `trtllm_spec_decode_drafted_tokens_total`/`_accepted_tokens_total` under per-request while the constructor creates `trtllm_spec_decode_num_draft_tokens_total`/`trtllm_spec_decode_num_accepted_tokens_total` - the docstring is stale on those two. `trtllm_kv_cache_utilization` is computed in the collector as kv_stats['usedNumBlocks'] / kv_stats['maxNumBlocks'], i.e. a block-count ratio from the engine's own stats, not bytes. Tracing: a `trace_span(name)` decorator in tensorrt_llm/llmapi/tracing.py opens SpanKind.SERVER spans around the serve path, gated on is_tracing_enabled(); note it is also used for insufficient_request_metrics_warning(), so the tracer doubles as a mechanism for flagging requests that arrived without enough metrics to evaluate - directly relevant to SLO coverage accounting. tracing_support set to per-request on 2026-10-04: a trace_span(name) decorator in tensorrt_llm/llmapi/tracing.py opens SpanKind.SERVER spans around the serve path, gated on is_tracing_enabled(), enabled via trtllm-serve --otlp_traces_endpoint (obs-trtllm-tracing-py, obs-trtllm-otel-readme). Not per-phase: the OTel guide covers proxy configuration for disaggregated serving so context propagates, but spans are not broken down by prefill vs decode. Unlike vLLM the packages are NOT bundled - the guide tells you to pip install opentelemetry-sdk, -api, -exporter-otlp and -semantic-conventions-ai yourself. Odd detail worth carrying: the same decorator also wraps insufficient_request_metrics_warning(), so the tracer doubles as a flag for requests that arrived without enough metrics to evaluate.

Notes

Metric list transcribed from the MetricsCollector class docstring and the constructor calls in tensorrt_llm/metrics/collector.py on main. Per-request: `trtllm_request_success_total` (label finished_reason), `trtllm_e2e_request_latency_seconds`, `trtllm_time_to_first_token_seconds`, `trtllm_time_per_output_token_seconds`, `trtllm_request_queue_time_seconds`, `trtllm_request_prefill_time_seconds`, `trtllm_request_decode_time_seconds`, `trtllm_request_inference_time_seconds`, `trtllm_prompt_tokens_total`, `trtllm_generation_tokens_total`, `trtllm_prompt_cached_tokens_total`, `trtllm_prompt_cache_hit_tokens_total` (labels cache_level, cache_tier), `trtllm_prompt_cached_tokens_per_request`, `trtllm_request_error_total`, `trtllm_prefill_perplexity`, `trtllm_generation_perplexity`. Iteration/scheduler: `trtllm_kv_cache_hit_rate`, `trtllm_kv_cache_utilization`, `trtllm_kv_cache_host_utilization`, `trtllm_kv_cache_iter_reuse_rate`, `trtllm_kv_cache_reused_blocks`, `trtllm_kv_cache_missed_blocks`, `trtllm_kv_cache_iter_reused_blocks`, `trtllm_kv_cache_iter_full_reused_blocks`, `trtllm_kv_cache_iter_partial_reused_blocks`, `trtllm_kv_cache_iter_missed_blocks`, `trtllm_kv_cache_gen_alloc_blocks_total`, `trtllm_kv_cache_onboard_bytes_total`, `trtllm_kv_cache_offload_bytes_total`, `trtllm_kv_cache_disk_prefetch_blocks_total`, `trtllm_kv_cache_intra_device_copy_bytes_total`, `trtllm_num_requests_running`, `trtllm_num_requests_waiting`, `trtllm_num_requests_completed_total`, `trtllm_max_num_active_requests`, `trtllm_iteration_latency_seconds`, `trtllm_gpu_memory_usage_bytes`, `trtllm_cpu_memory_usage_bytes`, `trtllm_pinned_memory_usage_bytes`, `trtllm_max_batch_size_static`, `trtllm_max_batch_size_runtime`, `trtllm_max_num_tokens_runtime`, `trtllm_kv_cache_max_blocks`, `trtllm_kv_cache_free_blocks`, `trtllm_kv_cache_used_blocks`, `trtllm_kv_cache_tokens_per_block`, `trtllm_num_context_requests`, `trtllm_num_generation_requests`, `trtllm_num_paused_requests`, `trtllm_num_scheduled_requests`, `trtllm_total_context_tokens`, `trtllm_avg_decoded_tokens_per_iter`. Spec decode: `trtllm_spec_decode_num_draft_tokens_total`, `trtllm_spec_decode_num_accepted_tokens_total`, `trtllm_spec_decode_acceptance_length`, `trtllm_spec_decode_draft_overhead`. Config-info gauges logged once at startup: `trtllm_model_config_info`, `trtllm_parallel_config_info`, `trtllm_speculative_config_info`, `trtllm_kv_cache_config_info`. Note the docstring lists `trtllm_spec_decode_drafted_tokens_total`/`_accepted_tokens_total` under per-request while the constructor creates `trtllm_spec_decode_num_draft_tokens_total`/`trtllm_spec_decode_num_accepted_tokens_total` - the docstring is stale on those two. `trtllm_kv_cache_utilization` is computed in the collector as kv_stats['usedNumBlocks'] / kv_stats['maxNumBlocks'], i.e. a block-count ratio from the engine's own stats, not bytes. Tracing: a `trace_span(name)` decorator in tensorrt_llm/llmapi/tracing.py opens SpanKind.SERVER spans around the serve path, gated on is_tracing_enabled(); note it is also used for insufficient_request_metrics_warning(), so the tracer doubles as a mechanism for flagging requests that arrived without enough metrics to evaluate - directly relevant to SLO coverage accounting. tracing_support set to per-request on 2026-10-04: a trace_span(name) decorator in tensorrt_llm/llmapi/tracing.py opens SpanKind.SERVER spans around the serve path, gated on is_tracing_enabled(), enabled via trtllm-serve --otlp_traces_endpoint (obs-trtllm-tracing-py, obs-trtllm-otel-readme). Not per-phase: the OTel guide covers proxy configuration for disaggregated serving so context propagates, but spans are not broken down by prefill vs decode. Unlike vLLM the packages are NOT bundled - the guide tells you to pip install opentelemetry-sdk, -api, -exporter-otlp and -semantic-conventions-ai yourself. Odd detail worth carrying: the same decorator also wraps insufficient_request_metrics_warning(), so the tracer doubles as a flag for requests that arrived without enough metrics to evaluate.

Cross-references

This record references no other records — it is a leaf. It is cited by 4.