LMDeploy time-per-output-token (TPOT) histogram
niche-lmdeploy-turbomind · histogram
Fields
id
metric-exposures/w3m-niche-lmdeploy-turbomind-lmdeploy-time-per-output-token-secondsMetric name
lmdeploy:time_per_output_token_seconds
Metric type
histogram
Unit
seconds
Exposure
prometheus
Endpoint
:23333/metrics
Histogram buckets
0.0050.010.0250.050.0750.10.150.20.30.40.50.7512.557.510204080
Labels
model_nameengine
Enabled by default
yes
Phase scoped
yes
notes
InternLM/lmdeploy main, PrometheusStatLogger in lmdeploy/metrics/loggers.py, read 2026-10-04. On by default and requires NO flag: the endpoint is simply available at the API server port (23333 by default). Label keys are model_name and engine on every metric, where engine carries the dp_rank. Bucket construction reuses vLLM's build_buckets / build_1_2_5_buckets idiom. Named time_PER_output_token (vLLM's is request_time_per_output_token_seconds) and its ladder starts at 0.005 rather than vLLM's 0.01, so it is NOT the same bucket list under a different name. Observed in record_iteration() from stats.tpot.
Notes
InternLM/lmdeploy main, PrometheusStatLogger in lmdeploy/metrics/loggers.py, read 2026-10-04. On by default and requires NO flag: the endpoint is simply available at the API server port (23333 by default). Label keys are model_name and engine on every metric, where engine carries the dp_rank. Bucket construction reuses vLLM's build_buckets / build_1_2_5_buckets idiom. Named time_PER_output_token (vLLM's is request_time_per_output_token_seconds) and its ladder starts at 0.005 rather than vLLM's 0.01, so it is NOT the same bucket list under a different name. Observed in record_iteration() from stats.tpot.