LMDeploy per-request cached-token histogram
niche-lmdeploy-turbomind · histogram
Fields
id
metric-exposures/w3m-niche-lmdeploy-turbomind-lmdeploy-request-cached-tokensMetric name
lmdeploy:request_cached_tokens
Metric type
histogram
Unit
tokens
Exposure
prometheus
Endpoint
:23333/metrics
Histogram buckets
1251020501002005001000200050008192
Labels
model_nameengine
Enabled by default
yes
notes
InternLM/lmdeploy main, PrometheusStatLogger in lmdeploy/metrics/loggers.py, read 2026-10-04. On by default and requires NO flag: the endpoint is simply available at the API server port (23333 by default). Label keys are model_name and engine on every metric, where engine carries the dp_rank. Bucket construction reuses vLLM's build_buckets / build_1_2_5_buckets idiom. Token-count histograms use build_1_2_5_buckets(max_model_len), so bounds depend on the served model's context length.
Notes
InternLM/lmdeploy main, PrometheusStatLogger in lmdeploy/metrics/loggers.py, read 2026-10-04. On by default and requires NO flag: the endpoint is simply available at the API server port (23333 by default). Label keys are model_name and engine on every metric, where engine carries the dp_rank. Bucket construction reuses vLLM's build_buckets / build_1_2_5_buckets idiom. Token-count histograms use build_1_2_5_buckets(max_model_len), so bounds depend on the served model's context length.