Argonne: GH200 x1, vLLM 0.6.1, Meta-Llama-3-8B FP16, 2048io, batch 1
267.8 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-gh200-vllm-metallama38b-fp16-b1-2048io-tpsMetric
tps_aggregate
Value
267.8
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Meta-Llama-3-8B
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). vLLM 0.6.1, BF16/FP16, 2048 in/2048 out, batch 1, 1x GH200 (96GB HBM3, Grace+Hopper superchip). Throughput = batch*(in+out)/end-to-end latency = AGGREGATE (batch-1 == per-request e2e rate incl prefill+decode). Reported latency n/as.
Measured by
third_party
Reproducible
yes