SGLang (observability surface: Prometheus metrics + OpenTelemetry with per-stage slices)

Metric names transcribed from SGLangMetricsCollector in python/sglang/srt/observability/metrics_collector.py on main (~180 name= occurrences, most conditional on feature flags). Core set: `sglang:num_running_reqs`, `sglang:num_queue_reqs`, `sglang:gen_throughput` ('The generation throughput (token/s)'), `sglang:cache_hit_rate`, `sglang:decode_sum_seq_lens`, `sglang:token_usage`, `sglang:full_token_usage`, `sglang:swa_token_usage`, `sglang:mamba_usage`, `sglang:num_used_tokens`, `sglang:kv_available_tokens`, `sglang:kv_evictable_tokens`, `sglang:kv_used_tokens`, `sglang:prompt_tokens_total`, `sglang:generation_tokens_total`, `sglang:cached_tokens_total`, `sglang:num_requests_total`, `sglang:num_aborted_requests_total`, `sglang:time_to_first_token_seconds` (label is_streaming), `sglang:inter_token_latency_seconds`, `sglang:request_time_per_output_token_seconds` (label is_streaming, documented as (finished_time - first_token_time) / (completion_tokens - 1)), `sglang:e2e_request_latency_seconds` (label is_streaming), `sglang:queue_time_seconds`, `sglang:per_stage_req_latency_seconds` (label stage), `sglang:uncached_prompt_tokens_histogram`, `sglang:prompt_tokens_histogram`, `sglang:generation_tokens_histogram`, `sglang:utilization`, `sglang:new_token_ratio`, `sglang:is_cuda_graph`, `sglang:max_total_num_tokens`, `sglang:kv_cache_memory_usage_gb`, `sglang:weight_memory_usage_gb`, `sglang:graph_memory_usage_gb{phase}`, `sglang:startup_time_seconds`, `sglang:page_size`, `sglang:num_pages`, `sglang:context_len`, `sglang:max_running_requests_under_SLO` (dead). The source itself flags `sglang:token_usage` as 'FIXME: misleadingly named' since it is the max across pools. Speculative decoding: `sglang:spec_accept_length`, `sglang:spec_accept_rate`, `sglang:spec_num_steps`, `sglang:spec_num_draft_tokens`, `sglang:spec_block_accept_length`. Also present and useful: `sglang:scheduler_idle_seconds_total`, `sglang:forward_execution_seconds_total`, `sglang:scheduler_stage_seconds_total`, `sglang:estimated_flops_per_gpu_total`, `sglang:estimated_read_bytes_per_gpu_total`. Trace shape: a root span named f'{role} Req {rid[:8]}' (role empty when unified, 'prefill'/'decode' otherwise) with attributes rid and module=sglang::{module_name}, plus bootstrap_room when set, over the incoming traceparent; then per-thread and per-slice spans named by RequestStage.stage_name at level 1-4. Stage names verbatim: tokenize, api_server_dispatch, dpc_dispatch, request_process, prefill_waiting, decode_forward, decode_loop, prefill_forward, chunked_prefill, mm_encode, prefill_prepare, prefill_bootstrap, prefill_transfer_kv_cache, decode_prepare, decode_bootstrap, decode_waiting, decode_transferred, fake_output, quick_finish, spec_draft, spec_verify, run_batch_cpu; mooncake adds mooncake_send, mooncake_recv, mooncake_worker_send, mooncake_worker_send_session, mooncake_worker_recv. TTFT definition from the docs is precise and worth quoting: 'time from request creation in the API server to the first token. For streaming requests it ends when the first output arrives after detokenization, which may contain no printable text' - so a detokenizer stall lands in TTFT. Retraction mechanics: when check_decode_mem() fails the scheduler logs 'KV cache pool is full. Retract requests. #retracted_reqs: N, #new_tokens_gained: M', calls release_kv_cache(is_insert=False) so the freed KV does not go back into the radix tree, and requeues; requests with no host backup are aborted with HTTPStatus.SERVICE_UNAVAILABLE rather than silently recomputed. tracing_support set to per-phase on 2026-10-04: SGLang is the only engine here whose trace is split by phase. Root span f'{role} Req {rid[:8]}' over per-thread and per-slice spans named from the RequestStage enum (tokenize, prefill_forward, decode_loop, decode_forward, and the disaggregation stages prefill_bootstrap, prefill_transfer_kv_cache, decode_prepare, decode_bootstrap, decode_waiting, decode_transferred), with nesting depth encoded as level 1-4. Evidence: obs-sglang-observability-trace-py, obs-sglang-request-stage-enum, and a 2-GPU CI test asserting a PD-mode request emits prefill_transfer_kv_cache, decode_transferred and mooncake_worker_send spans (obs-sglang-disaggregation-trace-test). Caveat: the OTel SDK is NOT bundled - process_tracing_init() raises RuntimeError unless opentelemetry is installed, so the surface exists in source but is absent from a default install.