sglang issue #3361 — community-measured DeepSeek-V3 FP8 on 2x8 NVIDIA H800 (TP16)
forum
Fields
id
sources/sglang-issue-3361-h800-deepseek-v3-fp8Publisher
sgl-project (GitHub issue, reporter @aooxin)
Kind
forum
Published
2025-02-07
Accessed
2026-10-15
notes
Community-reported sglang serving numbers on 2 nodes x 8 H800 80GB (TP16), DeepSeek-V3 FP8. Qianfan-14B dataset: avg input 216.79 tok, avg output 253.47 tok. Output-rate rows: 96 concurrent -> 622.05 tok/s, 64 -> 459.36, 32 -> 266.11 (avg TTFT 0.51-0.57s, e2e 30-38s). Measured by a third party on their own cluster; 'output rate' = generated tokens/sec aggregate across concurrent requests, not per-request decode. NOT vendor-measured; low-ish confidence but a real cluster run, not a spec.
Notes
Community-reported sglang serving numbers on 2 nodes x 8 H800 80GB (TP16), DeepSeek-V3 FP8. Qianfan-14B dataset: avg input 216.79 tok, avg output 253.47 tok. Output-rate rows: 96 concurrent -> 622.05 tok/s, 64 -> 459.36, 32 -> 266.11 (avg TTFT 0.51-0.57s, e2e 30-38s). Measured by a third party on their own cluster; 'output rate' = generated tokens/sec aggregate across concurrent requests, not per-request decode. NOT vendor-measured; low-ish confidence but a real cluster run, not a spec.
Cross-references
This record references no other records.