Qualcomm Cloud AI 100 Ultra, vLLM: Qwen2.5-32B peak aggregate tok/s (arXiv:2507.00418)
4926.6 tok/s aggregate (peak over 32-256 concurrent requests; 1 card)
Fields
id
benchmarks/arxiv2507-qaic100ultra-qwen2-5-32b-tps-aggMetric
tps_aggregate
Value
4,926.6
Unit
tok/s aggregate (peak over 32-256 concurrent requests; 1 card)
Model
Qwen2.5-32B
Methodology
THIRD-PARTY MEASURED (arXiv:2507.00418, PEARC'25, UCSD/NRP). vLLM serving, concurrency swept across 32/64/128/256 simultaneous requests dispatched via ThreadPoolExecutor to a local vLLM API; value is the PEAK AGGREGATE tokens/sec across the sweep (Table 2). Median latency/throughput computed per request set; coefficient of variation <5%. QAic = Cloud AI 100 Ultra, 1 card = 4 SoC. AGGREGATE throughput across concurrent requests, NOT per-request decode tok/s.
Measured by
third_party
Reproducible
yes
Power scope
device
Thermal state
steady
notes
Peak aggregate figure from a concurrency sweep — reads much higher than a single-stream decode rate and must not be compared to per-request tok/s. QAic QPC compile path (ONNX->QPC) is hardware-specific; numbers reflect the compiled deployment, not a naive run.
Notes
Peak aggregate figure from a concurrency sweep — reads much higher than a single-stream decode rate and must not be compared to per-request tok/s. QAic QPC compile path (ONNX->QPC) is hardware-specific; numbers reflect the compiled deployment, not a naive run.