Deploying DeepSeek on GB300 NVL72: Big Wins in Long-Context Inference (LMSYS Blog)

SGLang on GB300 NVL72 (72 GPUs) serving DeepSeek-R1-NVFP4 at 128K/8K ISL/OSL. Key TTFT result: 8.6s with dynamic chunking (32K initial chunk size). Without chunking TTFT exceeds 15s. Chunked prefill reduces TTFT by 30-45%; dynamic chunking provides further 8-17% reduction. Also reports 226.2 TPS/GPU peak throughput. Uses PD disaggregation, chunked PP, Wide-EP for decode, MTP, overlap scheduling, FP8 attention, FP8 KV cache, and GB300-specific hardware-accelerated softmax (2x SFU throughput).