inference/research
verifiedhigh · 0.92updated 2026-10-05

Dynamo frontend time-to-first-token histogram

nvidia-dynamo · histogram

Fields

id
metric-exposures/w5e-dynamo-dynamo-frontend-time-to-first-token-seconds
Metric name
dynamo_frontend_time_to_first_token_seconds
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
/metrics on the frontend service
Histogram buckets
00.00220.00470.010.0220.0470.10.220.4712.24.7102248100220480
Labels
model
Phase scoped
yes
notes
generate_log_buckets(0.001, 480.0, 18) - 18 generated bounds from 2.2 ms to 480 s. Override with DYN_METRICS_TTFT_{MIN,MAX,COUNT}. Attributes TTFT to the PREFILL worker in a disaggregated deployment (the collector's per-worker attribution logic does this explicitly) and simultaneously updates a per-worker gauge, so a reader can tell which half of a split deployment got slow. NVIDIA Dynamo main (ai-dynamo/dynamo), read 2026-10-05. IMPORTANT CONTEXT that changes how every record in this set reads: Dynamo is a serving framework that BRIDGES vendor engine registries rather than reimplementing them. components/src/dynamo/common/backend/metrics.py has engines call register_global_registry (default prometheus_client.REGISTRY) or register_engine_registry (a custom CollectorRegistry, e.g. SGLang's multiprocess one), and their vendor-prefixed series - vllm:, sglang:, trtllm_, lmcache: - appear on the SAME /metrics endpoint alongside Dynamo's own dynamo_* names. So on a Dynamo deployment you will find vLLM and SGLang metrics that are recorded in this repo against THOSE engines, and the dynamo_component / dynamo_namespace / dynamo_endpoint labels plus model and model_name are injected at scrape time by Dynamo. The names below are Dynamo's OWN, from the auto-generated prometheus_names.py (generated from lib/runtime/src/metrics/prometheus_names.rs - do not hand-edit). Prefix is dynamo_frontend.

Notes

generate_log_buckets(0.001, 480.0, 18) - 18 generated bounds from 2.2 ms to 480 s. Override with DYN_METRICS_TTFT_{MIN,MAX,COUNT}. Attributes TTFT to the PREFILL worker in a disaggregated deployment (the collector's per-worker attribution logic does this explicitly) and simultaneously updates a per-worker gauge, so a reader can tell which half of a split deployment got slow. NVIDIA Dynamo main (ai-dynamo/dynamo), read 2026-10-05. IMPORTANT CONTEXT that changes how every record in this set reads: Dynamo is a serving framework that BRIDGES vendor engine registries rather than reimplementing them. components/src/dynamo/common/backend/metrics.py has engines call register_global_registry (default prometheus_client.REGISTRY) or register_engine_registry (a custom CollectorRegistry, e.g. SGLang's multiprocess one), and their vendor-prefixed series - vllm:, sglang:, trtllm_, lmcache: - appear on the SAME /metrics endpoint alongside Dynamo's own dynamo_* names. So on a Dynamo deployment you will find vLLM and SGLang metrics that are recorded in this repo against THOSE engines, and the dynamo_component / dynamo_namespace / dynamo_endpoint labels plus model and model_name are injected at scrape time by Dynamo. The names below are Dynamo's OWN, from the auto-generated prometheus_names.py (generated from lib/runtime/src/metrics/prometheus_names.rs - do not hand-edit). Prefix is dynamo_frontend.

Cross-references