Argonne: GH200 x1, TensorRT-LLM 0.11.0, Llama-2-7b-hf FP16, 2048io, batch 1
334.6 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-gh200-trtllm-llama27bhf-fp16-b1-2048io-tpsMetric
tps_aggregate
Value
334.6
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Llama-2-7b-hf
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). TensorRT-LLM 0.11.0, BF16/FP16, 2048 in/2048 out, batch 1, 1x GH200 (96GB HBM3, Grace+Hopper superchip). Throughput = batch*(in+out)/end-to-end latency = AGGREGATE (batch-1 == per-request e2e rate incl prefill+decode). Reported latency n/as.
Measured by
third_party
Reproducible
yes
notes
TRT-LLM ~3x llama.cpp on same GH200 — CUDA-graph + fused-kernel path vs ggml.
Notes
TRT-LLM ~3x llama.cpp on same GH200 — CUDA-graph + fused-kernel path vs ggml.