SGLang on 8x RTX PRO 6000: prefix-cache hit on a 112K-token shared prefix serves in 0.34 s, ~39x the cold prefill rate

AGGREGATE-IS-NOT-APPLICABLE: this is a SINGLE REQUEST whose prompt was 112,000 tokens of already-cached prefix, served in 0.34 s wall clock. The tok/s figure is arithmetic on that (112,000 / 0.34 = ~329K; the source states ~325K), NOT a separately instrumented throughput counter - recorded as such so a reader does not mistake it for a measured rate. Same production configuration as the sibling records: 8x RTX PRO 6000 Blackwell Max-Q 96 GB (SM120, PCIe, no NVLink), TP=8/EP=8, kv_cache_dtype fp8_e4m3, max_running_requests 16, chunked_prefill_size 2048. The comparison figure in the same report is 6,853 tok/s for a 128K cold prefill, giving the ~39x.

Recorded primarily as the QUANTUM of the prefix-cache effect on a workload with heavy shared prefixes, which is the exact condition SGLang's RadixAttention exists for and the condition under which the engine's architecture pays for itself. Two honest limits. (1) The 39x is not a like-for-like engine comparison - it compares a cache hit against a cache miss ON THE SAME ENGINE, so it measures the cache, not the engine; any inference that SGLang is 39x faster than another engine from this number is wrong. (2) A 39x figure on a 112K-token prefix describes an unusually favourable workload shape (an agentic or multi-turn client that resends the same context), and it is sensitive to whether the prefix is byte-identical. The same report's llama.cpp counterpart - not the same project - is worth reading alongside: [[ev-llamacpp-spec-bench-replay-inflates]] documents how a similar cache-reuse path can make a BENCHMARK read 10x high rather than making a server 10x fast. LOAD-CONDITION AUDIT (bq- 2026-10-04). SCENARIO: NOT STATED by the source - absent, not inferred. A throughput figure with no scenario cannot be placed against an MLPerf Offline or Server number (they are different measurements; on identical hardware the Offline-minus-Server difference is negative in 5 of 9 systems measured). ABSENT CONDITIONS (2 of 7 checked: model, hardware, batch-or-concurrency, input length, output length, engine version, warmup): - output length: unrecoverable from ev-sglang-issue-40877-production-deepseek-v41-flash - would need the run's OSL or the harness's configured max_tokens - warmup: unrecoverable from ev-sglang-issue-40877-production-deepseek-v41-flash - would need the raw harness log's warmup phase or the benchmark command line (loadgen min_duration/min_query_count, or trtllm-bench --warm_up N / --num_runs N) None of these were filled in. A plausible value is worse than an absent one: it would be trusted. INTERCONNECT IDS FILLED 2026-10-04: interconnect_ids was empty; it now carries the interconnect list of the accelerator record(s) this run names (nvidia-rtx-pro-6000-blackwell). BASIS AND ITS LIMITS, stated because this field is easy to over-read: it records the FABRIC PRESENT IN THE HARDWARE CONFIGURATION, which is how every previously-filled record in this repo reads it, not traffic measured over that fabric. The clearest evidence for that convention is that single-GPU records already carry a scale-up fabric here - trtllm-llama31-8b-fp8-h100-1k1k-tps is 1x H100 at TP=1 and carries ['nvlink-4','nvswitch-4'] - and that host-link-only records carry a PCIe value (arc-pro-b70-mlperf-v6-1-llama2-70b-offline-tokens carries ['pcie-gen5'] on 4x Arc Pro B70, which has no GPU-to-GPU fabric at all). So a value here does NOT mean the interconnect was exercised by this particular workload, and must not be cited as evidence that it was. Whether the run actually crossed that fabric is a question about TP/EP/PP size, which this repo records in methodology, not here. NOT INVENTED: nothing was filled where the accelerator record itself carries no interconnect, so this field stays empty for any part whose own record has none - which is the honest state for a single-socket Apple sOC and for parts whose vendor publishes no host interface.