DeepSeek-V3 MTP speculative decoding in SGLang on 8x AMD MI300X: batch 64 latency speedup
1.25 speedup_ratio: 1.25x speedup over baseline without MTP (end-to-end latency reduced from 43848.20 ms to 35074.26 ms per request) at concurrency 64. BASELINE: DeepSeek-V3 served without MTP on identical 8x MI300X node in SGLang.
Fields
id
benchmarks/sd-deepseek-v3-mtp-sglang-mi300x-b64Metric
speedup_ratio
Value
1.25
Unit
speedup_ratio: 1.25x speedup over baseline without MTP (end-to-end latency reduced from 43848.20 ms to 35074.26 ms per request) at concurrency 64. BASELINE: DeepSeek-V3 served without MTP on identical 8x MI300X node in SGLang.
Model
deepseek-ai/DeepSeek-V3
Methodology
Concurrency (batch) = 64. Model: DeepSeek V3 with NextN MTP module as speculative drafter (--speculative-algo=NEXTN). Hardware: 8x AMD Instinct MI300X 192GB (Supermicro AS-8125GS-TNMR2, dual AMD EPYC 9654, ROCm 6.3.1). Engine: SGLang v0.5.0rc0. Dataset: Random dataset. Metric: End-to-end request latency (ms).
Measured by
vendor
Reproducible
yes
notes
Production serving figure: at batch/concurrency 64, the speedup compresses to 1.25x from 2.11x at batch 1 on Random dataset (and 1.36x at concurrency 64 on ShareGPT dataset). This demonstrates the classic batch dilution effect in production serving: as concurrency increases, GPU execution becomes compute-bound rather than memory-bandwidth-bound, reducing the relative benefit of speculative token verification.
Notes
Production serving figure: at batch/concurrency 64, the speedup compresses to 1.25x from 2.11x at batch 1 on Random dataset (and 1.36x at concurrency 64 on ShareGPT dataset). This demonstrates the classic batch dilution effect in production serving: as concurrency increases, GPU execution becomes compute-bound rather than memory-bandwidth-bound, reducing the relative benefit of speculative token verification.