Argonne: A100-40GB SXM x1, DeepSpeed-MII, Meta-Llama-3-8B FP16, 128io, batch 1
107.0 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-a100sxm40-nichedeepspeedmii-metallama38b-fp16-b1-128io-tpsMetric
tps_aggregate
Value
107
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Meta-Llama-3-8B
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136). DeepSpeed-MII, FP16, 128in/128out, batch 1, single A100 40GB SXM. Throughput = batch*(in+out)/e2e latency; batch-1 == per-request rate. Latency 2.393s.
Measured by
third_party
Reproducible
yes
notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).
Notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).