SGLang: DeepSeek 671B PD-disaggregated + large-scale EP on GB200 NVL72, 7583 decode tokens/s per GPU

Engine: SGLang with prefill-decode disaggregation and large-scale expert parallelism, using Blackwell DeepGEMM (FP8), DeepEP (NVLink-only), FlashInfer Blackwell FMHA for prefill, CUTLASS Blackwell MLA and Mooncake for KV transfer. Hardware: GB200 NVL72, 14 nodes available; 12 nodes (768 Blackwell GPUs) for decode, the remaining 2 for prefill. Model: DeepSeek 671B FP8. Prompt length 2000 tokens (the headline figure is specifically 'for 2,000-token input lengths'); output length NOT stated for the headline point - the ablation study elsewhere in the post uses input 2000 / output 100, which is probably but not certainly the headline configuration. Concurrency: NOT fixed and NOT stated - the setup deliberately uses a slowdown API to keep decode nodes saturated, so the operating point is 'saturated decode nodes' rather than a fixed batch size or user count. Tensor-parallel size: no single TP; large-scale EP is the dominant axis and TP16 vanilla is quoted only as the baseline. Figure is DECODE tokens/s per GPU, aggregate over concurrent requests, not per-request. Multi-token prediction (MTP) was NOT enabled for this figure (the post shows MTP only as a simulated baseline). Reported spread across sequence lengths: 2.5-3.4x vs H100 per GPU. Reproduction instructions referenced at sgl-project/sglang issue 7227.

SGLang is a third-party open-source project, not NVIDIA, so this is recorded as third_party even though the hardware was NVIDIA's and NVIDIA staff were acknowledged - the kernels being benchmarked (DeepGEMM, DeepEP, FlashInfer, CUTLASS) are largely NVIDIA-authored, which is why the number is unflattering to nobody in particular. The 2.7x-over-H100 comparison reuses baselines from the SGLang H100 post rather than re-measuring H100, so it is a cross-post comparison, not a same-session A/B. The post describes the work as 'early results' and 'preliminary'. CONVENTION AUDIT 2026-10-05 (agent w6-benchmarks): VALUE UNCHANGED. This record is CORRECTLY handled and is recorded here as the positive control for the sweep, because it is the case the per-user-vs-aggregate trap most easily breaks and it does not break. metric=decode_tok_s normally reads per-request, but this value is AGGREGATE decode throughput per GPU across the rack at saturation - a genuine mix of two normalizations on one key. Both facts are stated in the record itself (unit: "decode tokens/s per GPU (aggregate decode throughput per GPU across the rack)"; methodology: "Figure is DECODE tokens/s per GPU, aggregate over concurrent requests, not per-request"), so the record does the right thing the hard way. The remaining hazard is a reader filtering metric=decode_tok_s for single-user interactive decode: 7583 would be read as what one user gets, and the actual per-user figure on this deployment is not stated anywhere in the record - the concurrency is explicitly NOT fixed, the setup keeps decode nodes saturated by design, and no user count is given. So the number is honest and still unquotable as a latency figure. The general rule the sweep confirms across the corpus: a batch-1 record needs no convention statement because per-request and aggregate are the same number there, while EVERY record whose batch or concurrency exceeds 1 must say which one it is carrying - and the ones that fail to are enumerated in the sweep report, not fixed here.