inference/research
verifiedhigh · 0.95updated 2026-10-05

vLLM sampling-parameter max_tokens histogram

vllm · histogram

Fields

id
metric-exposures/w3m-vllm-vllm-request-params-max-tokens
Metric name
vllm:request_params_max_tokens
Metric type
histogram
Unit
tokens
Exposure
prometheus
Endpoint
:8000/metrics
Histogram buckets
125102050100200500100020005000
Labels
model_nameengine
Enabled by default
yes
notes
vLLM main, PrometheusStatLogger in vllm/v1/metrics/loggers.py, read 2026-10-04. On by default (EngineArgs.disable_log_stats defaults False); served at :8000/metrics. Bucket family name in --custom-histogram-buckets: request_tokens. Bucket bounds are build_1_2_5_buckets(max_model_len), so they DEPEND on the served model's context length; the list here is the ladder for a model with an 8192-token context. Bucket ladder re-verified against vllm/v1/metrics/buckets.py at v0.30.0 (main, 2026-09-22) on 2026-10-05: build_1_2_5_buckets(8192) returns [1,2,5,10,20,50,100,200,500,1000,2000,5000] - it stops at the first mantissa whose value exceeds max_value, the next candidate is 1*10^4=10000, which exceeds max_value, so the function returns immediately: 8192 is NEVER a bound at any context length. An earlier version of this record carried a trailing 8192, which is unreachable by construction and would make a histogram_quantile call silently wrong rather than absent.

Notes

vLLM main, PrometheusStatLogger in vllm/v1/metrics/loggers.py, read 2026-10-04. On by default (EngineArgs.disable_log_stats defaults False); served at :8000/metrics. Bucket family name in --custom-histogram-buckets: request_tokens. Bucket bounds are build_1_2_5_buckets(max_model_len), so they DEPEND on the served model's context length; the list here is the ladder for a model with an 8192-token context. Bucket ladder re-verified against vllm/v1/metrics/buckets.py at v0.30.0 (main, 2026-09-22) on 2026-10-05: build_1_2_5_buckets(8192) returns [1,2,5,10,20,50,100,200,500,1000,2000,5000] - it stops at the first mantissa whose value exceeds max_value, the next candidate is 1*10^4=10000, which exceeds max_value, so the function returns immediately: 8192 is NEVER a bound at any context length. An earlier version of this record carried a trailing 8192, which is unreachable by construction and would make a histogram_quantile call silently wrong rather than absent.

Cross-references