DeepSeek-V3 MTP speculative decoding in SGLang on 8x AMD MI300X: batch 1 latency speedup
2.11 speedup_ratio: 2.11x speedup over baseline without MTP (end-to-end latency reduced from 17348.25 ms to 8229.37 ms per request) at concurrency 1. BASELINE: DeepSeek-V3 served without MTP on identical 8x MI300X node in SGLang.
Fields
id
benchmarks/sd-deepseek-v3-mtp-sglang-mi300x-b1Metric
speedup_ratio
Value
2.11
Unit
speedup_ratio: 2.11x speedup over baseline without MTP (end-to-end latency reduced from 17348.25 ms to 8229.37 ms per request) at concurrency 1. BASELINE: DeepSeek-V3 served without MTP on identical 8x MI300X node in SGLang.
Model
deepseek-ai/DeepSeek-V3
Methodology
Concurrency (batch) = 1. Model: DeepSeek V3 with NextN MTP module as speculative drafter (--speculative-algo=NEXTN). Hardware: 8x AMD Instinct MI300X 192GB (Supermicro AS-8125GS-TNMR2, dual AMD EPYC 9654, ROCm 6.3.1). Engine: SGLang v0.5.0rc0. Dataset: Random dataset. Metric: End-to-end request latency (ms).
Measured by
vendor
Reproducible
yes
notes
Marketing/low-batch latency figure: at max concurrency 1 on the Random dataset, enabling MTP cuts end-to-end latency by 52.6% (17348.25 ms -> 8229.37 ms), achieving 2.11x speedup. Compare directly with [[sd-deepseek-v3-mtp-sglang-mi300x-b64]] to observe batch dilution: at batch/concurrency 64, the speedup drops to 1.25x (43848.20 ms -> 35074.26 ms).
Notes
Marketing/low-batch latency figure: at max concurrency 1 on the Random dataset, enabling MTP cuts end-to-end latency by 52.6% (17348.25 ms -> 8229.37 ms), achieving 2.11x speedup. Compare directly with [[sd-deepseek-v3-mtp-sglang-mi300x-b64]] to observe batch dilution: at batch/concurrency 64, the speedup drops to 1.25x (43848.20 ms -> 35074.26 ms).