vLLM bring-up of DeepSeek-V4.1-Flash on 8x RTX PRO 6000: 4584 tok/s prefill on the pure-torch eager fallback path at 1M context
Fields
benchmarks/ev-vllm-dsv41-production-128k-prefill-tps-eagerNotes
Compare with care against [[ev-sglang-dsv41-production-128k-prefill-tps]], which reports 6,853 tok/s for the same model family on the same 8x RTX PRO 6000 hardware. SGLang is ~1.5x this figure, and the honest reading is that MOST of that gap is probably the eager-vs-CUDA-graph and pure-torch-vs-flashinfer difference rather than the schedulers - the vLLM reporter states the fallback was forced by missing SM120 flashinfer kernels, not chosen. Both numbers are third-party single-site field reports, neither is a vendor benchmark, and neither is an independent reproduction of the other. The same vLLM report contains the more important correctness finding, which is why the deployment value matters: the FP8 KV-cache block layout is '[64tok x 128B fp8 | 64tok x 4B fp32 scale]' (block-appended, not per-token interleaved), and misreading it 'passes short-prompt smoke tests and only fails at >=100K needle retrieval - silent numerical corruption'. Verification method published: a synthetic ~1M-token document with two facts inserted at 50% and 80% depth. VRAM note: ~91 GB of 96 GB per GPU, with Engram tables dominating (2 x ~100 GB shards = 189 GiB across the TP group).
LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - output length: unrecoverable from ev-vllm-issue-56700-production-deepseek-v41-flash - would need the run's OSL or the harness's configured max_tokens - warmup: unrecoverable from ev-vllm-issue-56700-production-deepseek-v41-flash - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.