inference/research
verifiedhigh · 0.95updated 2026-10-05

SGLang KV-transfer volume histogram

sglang · histogram

Fields

id
metric-exposures/w3m-sglang-sglang-kv-transfer-total-mb
Metric name
sglang:kv_transfer_total_mb
Metric type
histogram
Unit
MB
Exposure
prometheus
Endpoint
:30000/metrics
Histogram buckets
1510501005001000500010000
Labels
model_nameengine_typetp_rankpp_rankmoe_ep_rank
Enabled by default
no
notes
SGLang main, python/sglang/srt/observability/metrics_collector.py, read 2026-10-04. NOT ON BY DEFAULT: nothing is exported unless --enable-metrics (ServerArgs default False), which is also the only thing that mounts /metrics on the server port. Label keys are model_name, engine_type, tp_rank, pp_rank, moe_ep_rank (plus dp_rank under data parallelism, priority under priority scheduling, and anything in extra_metric_labels). Bucket ladders are overridable per histogram via get_histogram_conf_from_env. Disaggregation-only: emitted and populated on a prefill/decode split deployment. On a unified deployment these are 0 rather than absent, so their presence is not evidence that the deployment is disaggregated. TYPE CORRECTED 2026-10-05: re-read of python/sglang/srt/observability/metrics_collector.py at v0.5.21 shows this is constructed as Histogram(name="sglang:kv_transfer_total_mb", documentation="Histogram of KV cache transfer size in MB.", buckets=(1, 5, 10, 50, 100, 500, 1000, 5000, 10000)). An earlier version of this record typed it 'gauge' with unit 'requests' and no buckets, which is the vLLM-style gauge shape rather than SGLang's - the name and unit were right, the type and buckets were not.

Notes

SGLang main, python/sglang/srt/observability/metrics_collector.py, read 2026-10-04. NOT ON BY DEFAULT: nothing is exported unless --enable-metrics (ServerArgs default False), which is also the only thing that mounts /metrics on the server port. Label keys are model_name, engine_type, tp_rank, pp_rank, moe_ep_rank (plus dp_rank under data parallelism, priority under priority scheduling, and anything in extra_metric_labels). Bucket ladders are overridable per histogram via get_histogram_conf_from_env. Disaggregation-only: emitted and populated on a prefill/decode split deployment. On a unified deployment these are 0 rather than absent, so their presence is not evidence that the deployment is disaggregated. TYPE CORRECTED 2026-10-05: re-read of python/sglang/srt/observability/metrics_collector.py at v0.5.21 shows this is constructed as Histogram(name="sglang:kv_transfer_total_mb", documentation="Histogram of KV cache transfer size in MB.", buckets=(1, 5, 10, 50, 100, 500, 1000, 5000, 10000)). An earlier version of this record typed it 'gauge' with unit 'requests' and no buckets, which is the vLLM-style gauge shape rather than SGLang's - the name and unit were right, the type and buckets were not.

Cross-references