inference/research
verifiedlow · 0.55updated 2026-10-15

sglang issue #3361 — community-measured DeepSeek-V3 FP8 on 2x8 NVIDIA H800 (TP16)

forum

Fields

id
sources/sglang-issue-3361-h800-deepseek-v3-fp8
Publisher
sgl-project (GitHub issue, reporter @aooxin)
Kind
forum
Published
2025-02-07
Accessed
2026-10-15
notes
Community-reported sglang serving numbers on 2 nodes x 8 H800 80GB (TP16), DeepSeek-V3 FP8. Qianfan-14B dataset: avg input 216.79 tok, avg output 253.47 tok. Output-rate rows: 96 concurrent -> 622.05 tok/s, 64 -> 459.36, 32 -> 266.11 (avg TTFT 0.51-0.57s, e2e 30-38s). Measured by a third party on their own cluster; 'output rate' = generated tokens/sec aggregate across concurrent requests, not per-request decode. NOT vendor-measured; low-ish confidence but a real cluster run, not a spec.

Notes

Community-reported sglang serving numbers on 2 nodes x 8 H800 80GB (TP16), DeepSeek-V3 FP8. Qianfan-14B dataset: avg input 216.79 tok, avg output 253.47 tok. Output-rate rows: 96 concurrent -> 622.05 tok/s, 64 -> 459.36, 32 -> 266.11 (avg TTFT 0.51-0.57s, e2e 30-38s). Measured by a third party on their own cluster; 'output rate' = generated tokens/sec aggregate across concurrent requests, not per-request decode. NOT vendor-measured; low-ish confidence but a real cluster run, not a spec.

Cross-references

This record references no other records.