sglang DeepSeek-V3 FP8 on 16x H800 (TP16): 266 tok/s aggregate @32 concurrent
266.11 tok/s generated (aggregate over 32 concurrent requests)
Fields
id
benchmarks/sglang-h800x16-deepseekv3-fp8-c32-tpsMetric
tps_aggregate
Value
266.11
Unit
tok/s generated (aggregate over 32 concurrent requests)
Model
deepseek-ai/DeepSeek-V3
Methodology
THIRD-PARTY MEASURED (sglang issue #3361, reporter @aooxin). SGLang launch_server, DeepSeek-V3 FP8, 2 nodes x 8 NVIDIA H800-80GB SXM, TP=16, --enable-torch-compile --torch-compile-max-bs 96 --disable-cuda-graph --quantization fp8. Qianfan-14B workload: avg prompt 216.79 tok, avg output 253.47 tok. 'Output rate' column = aggregate generated tok/s across N concurrent requests; NOT per-request decode. H800 = H100 with NVLink capped at 400 GB/s for the China export SKU. Config row: FP8 / max_bs=32 / concurrency=32.
Measured by
third_party
Reproducible
no
notes
Avg TTFT 0.51s, e2e 30.22s. NVLink interconnect = NVLink-4 (H100/H800 SXM5 generation; H800 caps it to 400 GB/s per the export SKU - see accelerator record).
Notes
Avg TTFT 0.51s, e2e 30.22s. NVLink interconnect = NVLink-4 (H100/H800 SXM5 generation; H800 caps it to 400 GB/s per the export SKU - see accelerator record).