Argonne: A100-40GB SXM x1, TensorRT-LLM 0.11.0, Llama-2-7b-hf FP16, 128io, batch 1
169.0 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-a100sxm40-tensorrtllm-llama27bhf-fp16-b1-128io-tpsMetric
tps_aggregate
Value
169
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Llama-2-7b-hf
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136). TensorRT-LLM 0.11.0, FP16, 128in/128out, batch 1, single A100 40GB SXM. Throughput = batch*(in+out)/e2e latency; batch-1 == per-request rate. Latency 1.515s.
Measured by
third_party
Reproducible
yes
notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).
Notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).