sglang DeepSeek-V3 FP8 on 16x H800 (TP16): 459 tok/s aggregate @64 concurrent
459.36 tok/s generated (aggregate over 64 concurrent requests)
Fields
id
benchmarks/sglang-h800x16-deepseekv3-fp8-c64-tpsMetric
tps_aggregate
Value
459.36
Unit
tok/s generated (aggregate over 64 concurrent requests)
Model
deepseek-ai/DeepSeek-V3
Methodology
THIRD-PARTY MEASURED (sglang issue #3361, reporter @aooxin). SGLang launch_server, DeepSeek-V3 FP8, 2 nodes x 8 NVIDIA H800-80GB SXM, TP=16, --enable-torch-compile --torch-compile-max-bs 96 --disable-cuda-graph --quantization fp8. Qianfan-14B workload: avg prompt 216.79 tok, avg output 253.47 tok. 'Output rate' column = aggregate generated tok/s across N concurrent requests; NOT per-request decode. H800 = H100 with NVLink capped at 400 GB/s for the China export SKU. Config row: FP8 / max_bs=64 / concurrency=64.
Measured by
third_party
Reproducible
no
notes
Avg TTFT 0.55s, e2e 34.83s. Same source row-family as the c96 record; kept separate because concurrency is part of the measured configuration. NVLink interconnect = NVLink-4 (H100/H800 SXM5 generation; H800 caps it to 400 GB/s per the export SKU - see accelerator record).
Notes
Avg TTFT 0.55s, e2e 34.83s. Same source row-family as the c96 record; kept separate because concurrency is part of the measured configuration. NVLink interconnect = NVLink-4 (H100/H800 SXM5 generation; H800 caps it to 400 GB/s per the export SKU - see accelerator record).