inference/research
draftmedium · 0.80updated 2026-10-04

SGLang serving DeepSeek-V4.1-Flash in production on 8x RTX PRO 6000: 128K cold prefill throughput at chunked_prefill_size=2048

6853.0 tok/s prefill (single stream, tokens counted from the response `usage` field, not SSE chunk counts)
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
benchmarks/ev-sglang-dsv41-production-128k-prefill-tps
Metric
prefill_tok_s
Value
6,853
Unit
tok/s prefill (single stream, tokens counted from the response `usage` field, not SSE chunk counts)
Model
DeepSeek-V4.1-Flash (V4.1 vision path, MXFP4 experts)
Methodology
AGGREGATE-IS-NOT-APPLICABLE: this is SINGLE-STREAM prefill throughput for one 128K-token prompt, measured cold, on a production configuration. Hardware: 8x NVIDIA RTX PRO 6000 Blackwell Max-Q (SM120, 96 GB each), PCIe only, NO NVLink. SGLang config as reported by the operator's own /get_server_info: tp_size 8, ep_size 8, dp_size 1, context_length 1,048,576, kv_cache_dtype fp8_e4m3, mem_fraction_static 0.72, max_running_requests 16, attention_backend dsv4, moe_runner_backend flashinfer_mxfp4, CUDA graph decode ON (max_bs 16) and PREFILL CUDA GRAPH DISABLED, chunked_prefill_size 2048, speculative_algorithm DSPARK with dspark_block_size 6. Container: docker run --gpus all --shm-size=64g --ipc=host --network host --restart unless-stopped. OMITTED BY SOURCE: SGLang version/commit, driver version, and whether the 128K prompt was cache-cold in the strict sense beyond the author's 'cold' label. REPRODUCIBILITY: the author published a full write-up with reproduction artifacts and an A/B sweep tool.
Measured by
third_party
Reproducible
yes
notes
Third-party field report from a production deployment, not a vendor benchmark and not a lab result - which is why it is in the repo at all, since it is one of only two engines in this dataset for which a named production site published its own numbers. The A/B the same report gives is worth more than the headline: re-running the identical config reads 6,765 tok/s (-1.3%), so run-to-run variance is ~1%. chunked_prefill_size sweep on the same hardware: 2048 -> 6,853; 4096 -> 6,301; 1024 -> 6,012 tok/s. Backend swap: flashmla prefill backend 6,845; fused-MoE all-reduce 6,828. A 32K prompt reads ~7,196 tok/s, i.e. longer prompts cost throughput. The DIAGNOSIS is the most transferable fact in the record: 'GPU util ~87% at 220 W of 600 W -> communication-bound on PCIe, not compute-bound', and '~7K tok/s looks like the ceiling for this board without NVLink'. 87% GPU utilization at 220 W of a 600 W cap is the signature of a link-bound rather than compute-bound run. ACCELERATOR CAVEAT: nvidia-rtx-pro-6000-blackwell is the closest accelerator id in this repo; the operator wrote 'RTX PRO 6000 Blackwell Max-Q, 96 GB' which is the workstation Max-Q SKU. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

Third-party field report from a production deployment, not a vendor benchmark and not a lab result - which is why it is in the repo at all, since it is one of only two engines in this dataset for which a named production site published its own numbers. The A/B the same report gives is worth more than the headline: re-running the identical config reads 6,765 tok/s (-1.3%), so run-to-run variance is ~1%. chunked_prefill_size sweep on the same hardware: 2048 -> 6,853; 4096 -> 6,301; 1024 -> 6,012 tok/s. Backend swap: flashmla prefill backend 6,845; fused-MoE all-reduce 6,828. A 32K prompt reads ~7,196 tok/s, i.e. longer prompts cost throughput. The DIAGNOSIS is the most transferable fact in the record: 'GPU util ~87% at 220 W of 600 W -> communication-bound on PCIe, not compute-bound', and '~7K tok/s looks like the ceiling for this board without NVLink'. 87% GPU utilization at 220 W of a 600 W cap is the signature of a link-bound rather than compute-bound run. ACCELERATOR CAVEAT: nvidia-rtx-pro-6000-blackwell is the closest accelerator id in this repo; the operator wrote 'RTX PRO 6000 Blackwell Max-Q, 96 GB' which is the workstation Max-Q SKU. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Interconnect (1)