Qualcomm Cloud AI 100 Ultra (16-card server): LLaMA 70B ISL=128 OSL=128 batch-64 throughput tok/s (vendor)
5700 tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)
Fields
id
benchmarks/qaic100ultra-llama70b-128-server-tpsMetric
tps_aggregate
Value
5,700
Unit
tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)
Model
LLaMA 70B
Methodology
VENDOR CLAIM (Qualcomm LLM Performance deck, Apr 2024). FP16 compute, MXFP6 weights. 16x Cloud AI 100 Ultra server, LLaMA 70B, input=128 output=128, throughput mode batch=64. Server-level aggregate tok/s across 16 accelerators - NOT per-card, NOT per-request decode.
Measured by
vendor
Reproducible
no
notes
Vendor slide-deck figure, batch=64 aggregate across a 16-card server. Companion latency-mode number: avg token latency <35 ms at batch=1. Reads high vs single-card numbers by 16x; do not compare against per-request decode.
Notes
Vendor slide-deck figure, batch=64 aggregate across a 16-card server. Companion latency-mode number: avg token latency <35 ms at batch=1. Reads high vs single-card numbers by 16x; do not compare against per-request decode.