inference/research
verifiedmedium · 0.80updated 2026-10-03

vLLM disaggregated PD serving: Qwen3.8-2.4T-A95B NVFP4 on GB300 NVL72, 180 generated tokens/s per user

180 generated tokens/s per user

Fields

id
benchmarks/vllm-qwen38-nvfp4-gb300-nvl72-interactivity
Metric
tok_s_per_user
Value
180
Unit
generated tokens/s per user
Model
Qwen3.8-2.4T-A95B-NVFP4
Methodology
Same deployment and build as the throughput sibling record: vLLM nightly-a9a17 on a GB300 NVL72 cluster (one NVLink72 domain), Qwen3.8-2.4T-A95B NVFP4, ISL/OSL 8192/1024, concurrency swept 1 to 2560, disaggregated PD with 1-4 prefill endpoints and one decode endpoint. 180 gen tok/s/user is the LOW-LATENCY (right-hand) end of the same Pareto frontier, i.e. it is achieved at much lower concurrency than the 5000 tok/s/GPU point - the exact concurrency for this endpoint is on a chart and is not stated in prose. Per-user figures come from GEN tokens/s divided by the number of concurrent users (trtllm/aiperf 'per user output throughput' convention), so it is a per-user rate, not per-request. Parallelism mixes TP/DP/EP. GSM8K 95%.
Measured by
vendor
Reproducible
yes
notes
Vendor-published on NVIDIA's own GB300 rack with fully public recipes, so the number is checkable even though the author is NVIDIA. Recorded as a separate record from the 5000 tok/s/GPU point because the two are different points on one curve at very different concurrency - quoting them as if they were one configuration would be wrong.

Notes

Vendor-published on NVIDIA's own GB300 rack with fully public recipes, so the number is checkable even though the author is NVIDIA. Recorded as a separate record from the 5000 tok/s/GPU point because the two are different points on one curve at very different concurrency - quoting them as if they were one configuration would be wrong.

Cross-references