SGLang Router (sgl-router)

SGLang's router is the routing layer for multi-model and data-parallel SGLang deployments. The data_parallel_controller.py implements four load balancing methods: ROUND_ROBIN (simple round-robin), FOLLOW_BOOTSTRAP_ROOM (route to the worker with the prefix cached), TOTAL_REQUESTS (route to the worker with the fewest requests), and TOTAL_TOKENS (route to the worker with the fewest total tokens). TOTAL_TOKENS is the most relevant for LLM serving because it is a proxy for KV cache pressure - a worker processing more tokens has less KV cache available. However, it is a proxy, not a direct measurement: it does not account for cache hits, and it does not account for the fact that a worker with many short requests may have more available cache than a worker with few long requests. The router is SGLang-specific and does not route to other engines. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2026-10-04 (0.0 months before this record), latest release v0.5.21 published 2026-10-02, 1996 contributors (distinct committers including anonymous), open_issues_count 5553 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 36777 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. UPDATED 2026-10-04 against the in-tree sgl-model-gateway README (dep-sgl-model-gateway-readme), which supersedes part of this record. The router has been generalised from a data-parallel controller into an explicit control plane plus data plane: control plane is the Worker Manager (validates workers, discovers capabilities, keeps the registry in sync), a Job Queue serialising add/remove with status at /workers/{worker_id}, a background health checker and load monitor, and optional Kubernetes service discovery; data plane is the HTTP/PD/gRPC/OpenAI routers plus a resilience layer (token-bucket rate limiting, request queuing, retry executor with default max 5 retries and exponential backoff triggered on 408/429/500/502/503/504, per-worker circuit breakers). Load-balancing policies are now enumerated: random, round_robin, cache_aware (a prefix tree of prompts with --cache-threshold, --balance-abs-threshold, --balance-rel-threshold, --eviction-interval, --max-tree-size), and power_of_two (lighter of two random candidates, integrated with LoadMonitor), with per-model overrides via --prefill-policy / --decode-policy in PD mode and via worker labels in IGW mode. Note what this does to the earlier FOLLOW_BOOTSTRAP_ROOM and TOTAL_TOKENS caveats: they are earlier-generation mechanisms that cache_aware and power_of_two supersede as the general path, though cache_aware carries the same class of staleness risk - see dep-kv-aware-router-view-cannot-be-rebuilt-after-restart for the upstream statement that the KV placement view cannot be rebuilt after a restart and degrades to worse routing rather than an error. Also newly documented: --enable-igw multi-model mode where workers are registered at runtime by POST /workers with url, model_id, priority and labels, which makes routing a mutable runtime API rather than static config. tracing_support=unknown (set 2026-10-04): this record shares sgl-project/sglang with obs-sglang-observability, whose tracing is per-phase (RequestStage-named slices such as prefill_forward, decode_loop and decode_transferred). That evidence is about the ENGINE, not the router, and the router binary's own tracing was not established from a source, so it is left unknown rather than inheriting per-phase. A reader who needs the router's span shape must read sgl-router's handler source. TRACING VERIFIED 2026-10-05. Read at main 2026-10-05 (v0.5.21). The Rust gateway crate sgl-model-gateway ships the tracing, not the Python launcher: otel_tracing_init(enable, otlp_endpoint) in src/observability/otel_trace.rs builds an OTLP-over-gRPC exporter (endpoint defaults to localhost:4317) with a TraceContextPropagator, so incoming W3C trace context is honoured. Spans are emitted under a three-target allowlist (smg::otel-trace, smg::observability::otel_trace, the events module) so anything raised elsewhere is silently dropped. middleware.rs's RequestSpan creates one 'http_request' span per HTTP request with fields method, uri, version, request_id, status_code, latency, error, module; request_id is declared Empty and filled by the later OnRequest handler because RequestIdLayer runs after TraceLayer. routers/grpc/common/stages/request_execution.rs adds a second 'grpc_generate' span carrying request_id, model and mode. Enabled by --enable-trace / --otlp-traces-endpoint. That is one span per request with a correlation id, with NO decomposition into prefill vs decode - a routing layer has no phases to decompose - hence per-request, not per-phase.