Qwen docs: Qwen3-8B FP8 on 1x H20 SGLang, input 1, 150.25 tok/s e2e
150.25 tok/s (batch-1 e2e, in=1+2048gen)
Fields
id
benchmarks/qwen-bench-h20-sglang-qwen3-8b-fp8-in1-tpsMetric
tps_aggregate
Value
150.25
Unit
tok/s (batch-1 e2e, in=1+2048gen)
Model
Qwen3-8B
Methodology
VENDOR-MEASURED (Qwen team, qwen.readthedocs.io Speed Benchmark). SGLang 0.4.6.post1 + sgl-kernel 0.1.0, PyTorch 2.6.0+cu124, vLLM 0.7.2 for AWQ-marlin. NVIDIA H20 96GB, context_length=140000, enable_mixed_chunk, mem_fraction_static=0.85, skip_tokenizer_init. Batch size 1, minimum GPU count. Speed=(prompt+generated)/e2e time = end-to-end per-request throughput including prefill — NOT pure decode. Generate 2048 tokens. Config: 1x H20.
Measured by
vendor
Reproducible
yes
notes
Metric is end-to-end (prompt+gen)/time on a single request — equivalent to ~decode rate only as input_len->0; at larger input lengths it blends prefill+decode. Reads LOWER than a pure tg number. Vendor page; H20 is the China-export Hopper (96GB HBM3, ~4.0 TB/s, 148 TFLOPS BF16).
Notes
Metric is end-to-end (prompt+gen)/time on a single request — equivalent to ~decode rate only as input_len->0; at larger input lengths it blends prefill+decode. Reads LOWER than a pure tg number. Vendor page; H20 is the China-export Hopper (96GB HBM3, ~4.0 TB/s, 148 TFLOPS BF16).