Argonne LLM-Inference-Bench: A100 40GB SXM x4, TensorRT-LLM, Llama-2-70b-hf, 128in/128out, batch 64
2525.97 tok/s aggregate across batch 64 (4x A100-40GB-SXM)
Fields
id
benchmarks/argonne-a100-tensorrtllm-llama2-70b-fp16-4x-b64-128io-tpsMetric
tps_aggregate
Value
2,525.97
Unit
tok/s aggregate across batch 64 (4x A100-40GB-SXM)
Model
meta-llama/Llama-2-70b-hf
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). TensorRT-LLM on NVIDIA A100 SXM 40GB, FP16/BF16 HF weights, 128 in / 128 out, batch 64, 4 GPU(s) (ThetaGPU 4xA100-40GB-SXM node; NVLink when n>1). End-to-end latency 6.486s. Throughput = batch*(in+out)/latency per paper Eq.2 — AGGREGATE over the batch, not per-request decode tok/s.
Measured by
third_party
Reproducible
yes
notes
Argonne ThetaGPU A100 rows are the 40GB SXM variant (not the 80GB SXM nor the PCIe card). Aggregate = batch*tokens/latency; at bs1 it equals single-request e2e throughput (prefill+decode mixed), NOT pure decode tok/s.
Notes
Argonne ThetaGPU A100 rows are the 40GB SXM variant (not the 80GB SXM nor the PCIe card). Aggregate = batch*tokens/latency; at bs1 it equals single-request e2e throughput (prefill+decode mixed), NOT pure decode tok/s.