inference/research
verifiedhigh · 0.94updated 2026-10-04

SGLang KV-pool token usage gauge (max across pools)

sglang · gauge

Fields

id
metric-exposures/w3m-sglang-sglang-token-usage
Metric name
sglang:token_usage
Metric type
gauge
Unit
1
Exposure
prometheus
Endpoint
:30000/metrics
Labels
model_nameengine_typetp_rankpp_rankmoe_ep_rank
Enabled by default
no
notes
SGLang main, python/sglang/srt/observability/metrics_collector.py, read 2026-10-04. NOT ON BY DEFAULT: nothing is exported unless --enable-metrics (ServerArgs default False), which is also the only thing that mounts /metrics on the server port. Label keys are model_name, engine_type, tp_rank, pp_rank, moe_ep_rank (plus dp_rank under data parallelism, priority under priority scheduling, and anything in extra_metric_labels). Bucket ladders are overridable per histogram via get_histogram_conf_from_env. The source itself flags this one: 'FIXME: misleadingly named; rename requires API deprecation'. It is the MAX across the full, SWA and mamba pools, not a single cache's occupancy. For per-pool occupancy use sglang:full_token_usage, sglang:swa_token_usage and sglang:mamba_usage.

Notes

SGLang main, python/sglang/srt/observability/metrics_collector.py, read 2026-10-04. NOT ON BY DEFAULT: nothing is exported unless --enable-metrics (ServerArgs default False), which is also the only thing that mounts /metrics on the server port. Label keys are model_name, engine_type, tp_rank, pp_rank, moe_ep_rank (plus dp_rank under data parallelism, priority under priority scheduling, and anything in extra_metric_labels). Bucket ladders are overridable per histogram via get_histogram_conf_from_env. The source itself flags this one: 'FIXME: misleadingly named; rename requires API deprecation'. It is the MAX across the full, SWA and mamba pools, not a single cache's occupancy. For per-pool occupancy use sglang:full_token_usage, sglang:swa_token_usage and sglang:mamba_usage.

Cross-references