Argonne: A100-40GB SXM x1, TensorRT-LLM 0.11.0, Meta-Llama-3-8B FP16, 128io, batch 1
143.7 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-a100sxm40-tensorrtllm-metallama38b-fp16-b1-128io-tpsMetric
tps_aggregate
Value
143.7
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Meta-Llama-3-8B
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136). TensorRT-LLM 0.11.0, FP16, 128in/128out, batch 1, single A100 40GB SXM. Throughput = batch*(in+out)/e2e latency; batch-1 == per-request rate. Latency 1.781s.
Measured by
third_party
Reproducible
yes
notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).
Notes
A100 40GB SXM (ANL Polaris-class node, 4x per node; this row is single-device).