Qualcomm Cloud AI 100 Ultra (16-card server): LLaMA 7B ISL=2048 OSL=2048 batch-64 throughput tok/s (vendor)
7300 tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)
Fields
id
benchmarks/qaic100ultra-llama7b-2048-server-tpsMetric
tps_aggregate
Value
7,300
Unit
tok/s server aggregate (16x AI 100 Ultra, throughput mode batch=64)
Model
LLaMA 7B
Methodology
VENDOR CLAIM (Qualcomm LLM Performance deck, Apr 2024). FP16 compute, MXFP6 weights. 16x Cloud AI 100 Ultra server, LLaMA 7B, input=2048 output=2048, throughput mode batch=64. Server-level aggregate tok/s across 16 accelerators - NOT per-card, NOT per-request decode.
Measured by
vendor
Reproducible
no
notes
Vendor slide-deck figure, batch=64 aggregate across a 16-card server. Companion latency-mode number: avg token latency <10 ms at batch=1. Reads high vs single-card numbers by 16x; do not compare against per-request decode.
Notes
Vendor slide-deck figure, batch=64 aggregate across a 16-card server. Companion latency-mode number: avg token latency <10 ms at batch=1. Reads high vs single-card numbers by 16x; do not compare against per-request decode.