sglang DeepSeek-V3 FP8 on 16x H800 (TP16, 2 nodes): 622 tok/s aggregate @96 concurrent
622.05 tok/s generated (aggregate over 96 concurrent requests, avg 253 output tok each)
Fields
id
benchmarks/sglang-h800x16-deepseekv3-fp8-c96-tpsMetric
tps_aggregate
Value
622.05
Unit
tok/s generated (aggregate over 96 concurrent requests, avg 253 output tok each)
Model
deepseek-ai/DeepSeek-V3
Methodology
THIRD-PARTY MEASURED (sglang issue #3361, reporter @aooxin). SGLang launch_server, DeepSeek-V3 FP8, 2 nodes x 8 NVIDIA H800-80GB SXM, TP=16, --enable-torch-compile --torch-compile-max-bs 96 --disable-cuda-graph --quantization fp8. Qianfan-14B workload: avg prompt 216.79 tok, avg output 253.47 tok. 'Output rate' column = aggregate generated tok/s across N concurrent requests; NOT per-request decode. H800 = H100 with NVLink capped at 400 GB/s for the China export SKU. Config row: FP8 / max_bs=96 / concurrency=96.
Measured by
third_party
Reproducible
no
notes
Avg TTFT 0.57s, avg e2e sentence latency 38.30s in this row. Per-request decode ~622/96 = 6.5 tok/s/user at saturation. Reporter could not match 'official' numbers and asked the SGLang team for tuning help - treat as a lower-bound achievable figure, not a tuned ceiling. First benchmark coverage for nvidia-h800-80gb. NVLink interconnect = NVLink-4 (H100/H800 SXM5 generation; H800 caps it to 400 GB/s per the export SKU - see accelerator record).
Notes
Avg TTFT 0.57s, avg e2e sentence latency 38.30s in this row. Per-request decode ~622/96 = 6.5 tok/s/user at saturation. Reporter could not match 'official' numbers and asked the SGLang team for tuning help - treat as a lower-bound achievable figure, not a tuned ceiling. First benchmark coverage for nvidia-h800-80gb. NVLink interconnect = NVLink-4 (H100/H800 SXM5 generation; H800 caps it to 400 GB/s per the export SKU - see accelerator record).