Argonne: A100-40GB SXM x1, DeepSpeed-MII, Mistral-7B-v0.1 FP16, 128io, batch 1
116.3 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-a100sxm40-nichedeepspeedmii-mistral7bv01-fp16-b1-128io-tpsMetric
tps_aggregate
Value
116.3
Unit
tok/s aggregate (batch 1, single request)
Model
mistralai/Mistral-7B-v0.1
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136). DeepSpeed-MII, FP16, 128in/128out, batch 1, single A100 40GB SXM. Throughput = batch*(in+out)/e2e latency; batch-1 == per-request rate. Latency 2.201s.
Measured by
third_party
Reproducible
yes
notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).
Notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).