inference/research
draftmedium · 0.75updated 2026-10-04

SGLang serving DeepSeek-V4.1-Flash in production on 8x RTX PRO 6000: single-stream decode throughput, prose vs structured output

93.9 tok/s decode, single stream (one user at a time)
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
benchmarks/ev-sglang-dsv41-production-decode-tok-s-per-user
Metric
tok_s_per_user
Value
93.9
Unit
tok/s decode, single stream (one user at a time)
Model
DeepSeek-V4.1-Flash
Methodology
PER-REQUEST, NOT AGGREGATE: one concurrent user, production configuration, TP=8/EP=8 across 8x RTX PRO 6000 Blackwell Max-Q 96 GB (SM120, PCIe, no NVLink), kv_cache_dtype fp8_e4m3, mem_fraction_static 0.72, max_running_requests 16, attention_backend dsv4, moe_runner_backend flashinfer_mxfp4, speculative_algorithm DSPARK dspark_block_size 6, CUDA graph decode on. Measured through a client path over an SSH tunnel. TTFT 143-169 ms on the same runs. TWO PROMPT CLASSES, because the split is the finding: prose 93.9 tok/s, structured/JSON output 134.1 tok/s - structured generation is 43% FASTER, which is the opposite of what a naive expectation would predict and is a direct consequence of speculative decoding accepting more tokens when the continuation is schema-constrained.
Measured by
third_party
Reproducible
yes
notes
This is the number a reader should compare against for 'what does production decode feel like on consumer Blackwell with no NVLink', and the accompanying caveat is that the hardware is the constraint: the same report measures 87% GPU utilization at 220 W of 600 W and calls it communication-bound on PCIe. SGLang version/commit, driver version and the SSH-tunnel bandwidth are OMITTED BY SOURCE, and the author explicitly says the numbers 'come from a working day of production use' with 'No 24 h soak'. So this is a single-day production sample, not a characterised steady state. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Notes

This is the number a reader should compare against for 'what does production decode feel like on consumer Blackwell with no NVLink', and the accompanying caveat is that the hardware is the constraint: the same report measures 87% GPU utilization at 220 W of 600 W and calls it communication-bound on PCIe. SGLang version/commit, driver version and the SSH-tunnel bandwidth are OMITTED BY SOURCE, and the author explicitly says the numbers 'come from a working day of production use' with 'No 24 h soak'. So this is a single-day production sample, not a characterised steady state. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.

Cross-references

Accelerators (1)

Engine (1)

Interconnect (1)