Argonne LLM-Inference-Bench: A100 40GB SXM x1, vLLM, Llama-2-7b-hf, 128in/128out, batch 1
154.87 tok/s aggregate across batch 1 (1x A100-40GB-SXM)
Fields
id
benchmarks/argonne-a100-vllm-llama2-7b-fp16-b1-128io-tpsMetric
tps_aggregate
Value
154.87
Unit
tok/s aggregate across batch 1 (1x A100-40GB-SXM)
Model
meta-llama/Llama-2-7b-hf
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). vLLM on NVIDIA A100 SXM 40GB, FP16/BF16 HF weights, 128 in / 128 out, batch 1, 1 GPU(s) (ThetaGPU 4xA100-40GB-SXM node; NVLink when n>1). End-to-end latency 1.653s. Throughput = batch*(in+out)/latency per paper Eq.2 — AGGREGATE over the batch, not per-request decode tok/s.
Measured by
third_party
Reproducible
yes
notes
Argonne ThetaGPU A100 rows are the 40GB SXM variant (not the 80GB SXM nor the PCIe card). Aggregate = batch*tokens/latency; at bs1 it equals single-request e2e throughput (prefill+decode mixed), NOT pure decode tok/s.
Notes
Argonne ThetaGPU A100 rows are the 40GB SXM variant (not the 80GB SXM nor the PCIe card). Aggregate = batch*tokens/latency; at bs1 it equals single-request e2e throughput (prefill+decode mixed), NOT pure decode tok/s.