inference/research
verifiedhigh · 0.94updated 2026-10-04

SGLang per-request time-per-output-token (TPOT) histogram

sglang · histogram

Fields

id
metric-exposures/w3m-sglang-sglang-request-time-per-output-token-seconds
Metric name
sglang:request_time_per_output_token_seconds
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:30000/metrics
Histogram buckets
0.0020.0040.0060.0080.010.0150.020.0250.030.0350.040.060.080.10.20.40.60.812468
Labels
model_nameengine_typetp_rankpp_rankmoe_ep_rankis_streaming
Enabled by default
no
Phase scoped
yes
notes
SGLang main, python/sglang/srt/observability/metrics_collector.py, read 2026-10-04. NOT ON BY DEFAULT: nothing is exported unless --enable-metrics (ServerArgs default False), which is also the only thing that mounts /metrics on the server port. Label keys are model_name, engine_type, tp_rank, pp_rank, moe_ep_rank (plus dp_rank under data parallelism, priority under priority scheduling, and anything in extra_metric_labels). Bucket ladders are overridable per histogram via get_histogram_conf_from_env. Source documentation states the definition verbatim: 'Per-request TPOT in seconds: (finished_time - first_token_time) / (completion_tokens - 1)'. Shared bucket ladder with the ITL histogram; the denominators differ, so the quantiles are not the same statistic.

Notes

SGLang main, python/sglang/srt/observability/metrics_collector.py, read 2026-10-04. NOT ON BY DEFAULT: nothing is exported unless --enable-metrics (ServerArgs default False), which is also the only thing that mounts /metrics on the server port. Label keys are model_name, engine_type, tp_rank, pp_rank, moe_ep_rank (plus dp_rank under data parallelism, priority under priority scheduling, and anything in extra_metric_labels). Bucket ladders are overridable per histogram via get_histogram_conf_from_env. Source documentation states the definition verbatim: 'Per-request TPOT in seconds: (finished_time - first_token_time) / (completion_tokens - 1)'. Shared bucket ladder with the ITL histogram; the denominators differ, so the quantiles are not the same statistic.

Cross-references