inference/research
draftmedium · 0.75updated 2026-10-04

SGLang RadixAttention hit on 8x RTX PRO 6000: 112K-token shared prefix served in 0.34 s (~39x the cold prefill rate)

This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
kv-efficiency/sglang-radix-hit-112k-prefix
Model
DeepSeek-V4.1-Flash (MXFP4 experts, 1M context enabled)
Context length
114,688
KV (bytes/token)
35,136
KV bytes basis
61 layers x (kv_lora_rank 512 + qk_rope_head_dim 64) x 1 byte (fp8_e4m3) = 61 x 576 = 35,136 B/token, same MLA latent skeleton as sglang-dsv41-fp8kv-128k-cold-prefill. On a HIT the resident cost is not paid again, which is the entire mechanism: the radix tree maps the 112K-token prefix to already-resident KV blocks instead of recomputing them.
KV dtype
fp8
Attention variant
mla
Prefix reuse
radix
TTFT (ms) @ ctx
340
tok/s @ ctx
325,000
Methodology
SINGLE REQUEST whose 112,000-token prompt was an already-cached shared prefix on the same production SGLang config as sglang-dsv41-fp8kv-128k-cold-prefill (8x RTX PRO 6000, TP=8/EP=8, kv_cache_dtype fp8_e4m3, RadixAttention). measured_tps_at_ctx is arithmetic on the wall clock (112,000 / 0.34 s ~ 329K; the source states ~325K), i.e. a prefill-equivalent rate on a cache HIT, not a separately instrumented throughput counter — recorded because the source publishes it as the comparison to the 6,853 tok/s cold figure (~39x).
Measured by
third_party
Reproducible
yes
notes
THE HEADLINE TRAP, stated because this is the most misquotable number in the type: ~39x is a property of the CACHE, not of SGLang vs any other engine — it compares a radix hit against a cold miss on the same engine, so 'SGLang is 39x faster' is the wrong reading in both directions. The honest use is the quantum of prefix reuse on a workload that resends long shared prefixes (agentic / multi-turn), which is exactly the condition RadixAttention exists for. Requires a byte-identical prefix; near-prefixes do not hit. Compare [[benchmarks/ev-llamacpp-spec-bench-replay-inflates]], where a similar replay path makes a BENCHMARK read ~13x high rather than making a server fast — the distinction this record's notes exist to keep straight. measured_ttft_ms_at_ctx = 340 ms is the wall-clock serve time the source reports for the hit.

Notes

THE HEADLINE TRAP, stated because this is the most misquotable number in the type: ~39x is a property of the CACHE, not of SGLang vs any other engine — it compares a radix hit against a cold miss on the same engine, so 'SGLang is 39x faster' is the wrong reading in both directions. The honest use is the quantum of prefix reuse on a workload that resends long shared prefixes (agentic / multi-turn), which is exactly the condition RadixAttention exists for. Requires a byte-identical prefix; near-prefixes do not hit. Compare [[benchmarks/ev-llamacpp-spec-bench-replay-inflates]], where a similar replay path makes a BENCHMARK read ~13x high rather than making a server fast — the distinction this record's notes exist to keep straight. measured_ttft_ms_at_ctx = 340 ms is the wall-clock serve time the source reports for the hit.

Cross-references

Accelerators (1)

Engine (1)

Interconnect (1)