SGLang on GB300 NVL72: DeepSeek-R1-NVFP4 TTFT 8.6s at 128K/8K ISL/OSL with dynamic chunking

Workload: 128K input / 8K output (ISL/OSL). Hardware: GB300 NVL72 (72 Blackwell Ultra GPUs, 288 GB HBM3e per GPU). Engine: SGLang with PD disaggregation, chunked pipeline parallelism (PP), Wide-EP for decode, MTP, overlap scheduling, FP8 attention, FP8 KV cache, GB300 hardware-accelerated softmax (2x SFU throughput). Prefill: dynamic chunking with initial chunk size 32K. Concurrency: not explicitly stated (likely moderate to high given throughput context). Cache state: cold (no prefix caching mentioned; long-context RAG workload). Without chunking TTFT exceeds 15s; chunked prefill reduces TTFT by 30-45%; dynamic chunking provides further 8-17% reduction.

Published by LMSYS/NVIDIA on 2026-02-19. Reproduction instructions at sgl-project/sglang issue 18703. The 8.6s TTFT for 128K prefill on 72 Blackwell Ultra GPUs represents the state of the art for long-context prefill on the most powerful available hardware. Key insight: even with 72 GPUs and aggressive optimization (chunked PP, dynamic chunking, FP8 KV, hardware softmax), 128K prefill still takes 8.6s. The FMHA kernel itself takes 205ms on GB300 vs 277ms on GB200 (1.35x speedup from 2x SFU throughput). Also reports 226.2 TPS/GPU peak throughput under the same workload. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - engine version: unrecoverable from bench-lmsys-gb300-longctx-ttft - engine_id=sglang is named but no build/commit/container digest is published - warmup: unrecoverable from bench-lmsys-gb300-longctx-ttft - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. FORMAT_ID BACKFILL 2026-10-08 (w10-bench-formatid): format_id was null and is now 'nvfp4'. EVIDENCE, quoted from this record's own text: model: "DeepSeek-R1 (NVFP4)". NO NUMBER CHANGED - value, metric and unit are untouched; this adds metadata, it does not re-measure. The value was NOT taken from a model name, from a sibling record, or from outside knowledge. Left null everywhere the record's own unit/methodology does not name a format for this run: 63 of the 150 records in data/benchmarks/ are still null and each one has its reason, because a null with a reason is correct and a guessed value is not.