inference/research
verifiedhigh · 0.90updated 2026-10-04

TensorRT-LLM llm_args.py - return_perf_metrics default and PrometheusMetricsConfig

repo

Fields

id
sources/w3m-trtllm-return-perf-metrics-default
Publisher
NVIDIA/TensorRT-LLM
Kind
repo
Accessed
2026-10-04
notes
tensorrt_llm/llmapi/llm_args.py on main, 2026-10-04. `return_perf_metrics: bool = Field(default=False, description="Allow serving responses to include per-request performance metrics when the request sets X-TRTLLM-return-metrics: 1.", status="prototype")` - default False, and per the openai_server.py source comment this is what also mounts the Prometheus endpoint. `per_request_spec_decode_stats` is deliberately independent of it. `prometheus_metrics_config: Optional[PrometheusMetricsConfig] = Field(default=None, ...)`, where PrometheusMetricsConfig (line 3436) carries e2e_request_latency_buckets, time_to_first_token_buckets, time_per_output_token_buckets, request_queue_time_buckets, request_prefill_time_buckets, request_decode_time_buckets and request_inference_time_buckets. So every TRT-LLM latency histogram's buckets are overridable, and all seven default to the library literals when unset.

Notes

tensorrt_llm/llmapi/llm_args.py on main, 2026-10-04. `return_perf_metrics: bool = Field(default=False, description="Allow serving responses to include per-request performance metrics when the request sets X-TRTLLM-return-metrics: 1.", status="prototype")` - default False, and per the openai_server.py source comment this is what also mounts the Prometheus endpoint. `per_request_spec_decode_stats` is deliberately independent of it. `prometheus_metrics_config: Optional[PrometheusMetricsConfig] = Field(default=None, ...)`, where PrometheusMetricsConfig (line 3436) carries e2e_request_latency_buckets, time_to_first_token_buckets, time_per_output_token_buckets, request_queue_time_buckets, request_prefill_time_buckets, request_decode_time_buckets and request_inference_time_buckets. So every TRT-LLM latency histogram's buckets are overridable, and all seven default to the library literals when unset.

Cross-references

This record references no other records.