Argonne: GH200 x1, llama.cpp, Meta-Llama-3-8B FP16, 2048io, batch 1
100.5 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-gh200-llamacpp-metallama38b-fp16-b1-2048io-tpsMetric
tps_aggregate
Value
100.5
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Meta-Llama-3-8B
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). llama.cpp (ggml CUDA build), BF16/FP16, 2048 in/2048 out, batch 1, 1x GH200 (96GB HBM3, Grace+Hopper superchip). Throughput = batch*(in+out)/end-to-end latency = AGGREGATE (batch-1 == per-request e2e rate incl prefill+decode). Reported latency n/as.
Measured by
third_party
Reproducible
yes
notes
GH200 = Grace CPU + Hopper GPU superchip; GPU half is H100-96GB-class. batch-1 e2e throughput.
Notes
GH200 = Grace CPU + Hopper GPU superchip; GPU half is H100-96GB-class. batch-1 e2e throughput.