Argonne: GH200 x1, llama.cpp, Qwen2-7B FP16, 2048io, batch 1
104.6 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-gh200-llamacpp-qwen27b-fp16-b1-2048io-tpsMetric
tps_aggregate
Value
104.6
Unit
tok/s aggregate (batch 1, single request)
Model
Qwen/Qwen2-7B
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). llama.cpp (ggml CUDA build), BF16/FP16, 2048 in/2048 out, batch 1, 1x GH200 (96GB HBM3, Grace+Hopper superchip). Throughput = batch*(in+out)/end-to-end latency = AGGREGATE (batch-1 == per-request e2e rate incl prefill+decode). Reported latency n/as.
Measured by
third_party
Reproducible
yes
notes
GH200 = Grace CPU + Hopper GPU superchip; GPU half is H100-96GB-class. batch-1 e2e throughput.
Notes
GH200 = Grace CPU + Hopper GPU superchip; GPU half is H100-96GB-class. batch-1 e2e throughput.