Qwen docs: Qwen3-235B-A22B FP8 on 4x H20 SGLang, input 30720, 887.9 tok/s e2e
887.9 tok/s (batch-1 e2e, in=30720+2048gen, TP=4)
Fields
id
benchmarks/qwen-bench-h20-sglang-qwen3-235b-a22b-fp8-in30720-tpsMetric
tps_aggregate
Value
887.9
Unit
tok/s (batch-1 e2e, in=30720+2048gen, TP=4)
Model
Qwen3-235B-A22B
Methodology
VENDOR-MEASURED (Qwen team, qwen.readthedocs.io Speed Benchmark). SGLang 0.4.6.post1 + sgl-kernel 0.1.0, PyTorch 2.6.0+cu124, vLLM 0.7.2 for AWQ-marlin. NVIDIA H20 96GB, context_length=140000, enable_mixed_chunk, mem_fraction_static=0.85, skip_tokenizer_init. Batch size 1, minimum GPU count. Speed=(prompt+generated)/e2e time = end-to-end per-request throughput including prefill — NOT pure decode. Generate 2048 tokens. Config: 4x H20, TP=4.
Measured by
vendor
Reproducible
yes
notes
Metric is end-to-end (prompt+gen)/time on a single request — equivalent to ~decode rate only as input_len->0; at larger input lengths it blends prefill+decode. Reads LOWER than a pure tg number. Vendor page; H20 is the China-export Hopper (96GB HBM3, ~4.0 TB/s, 148 TFLOPS BF16). TP=4 across 4 H20 96GB cards.
Notes
Metric is end-to-end (prompt+gen)/time on a single request — equivalent to ~decode rate only as input_len->0; at larger input lengths it blends prefill+decode. Reads LOWER than a pure tg number. Vendor page; H20 is the China-export Hopper (96GB HBM3, ~4.0 TB/s, 148 TFLOPS BF16). TP=4 across 4 H20 96GB cards.