Argonne LLM-Inference-Bench: MI250 x1, llama.cpp (ROCm/HIP), Llama-2-7B, 128in/128out, batch 1
44.4 tok/s aggregate (batch 1, single request)
Fields
id
benchmarks/argonne-mi250-llamacpp-llama2-7b-fp16-b1-128io-tpsMetric
tps_aggregate
Value
44.4
Unit
tok/s aggregate (batch 1, single request)
Model
meta-llama/Llama-2-7b-hf
Methodology
THIRD-PARTY MEASURED (Argonne LLM-Inference-Bench, arXiv:2411.00136, All_results.csv). llama.cpp HIP backend, FP16 weights via HuggingFace checkpoint (not GGUF-quantized), 128 in / 128 out, batch 1. Latency 5.762s. AGGREGATE = single-request throughput at bs1.
Measured by
third_party
Reproducible
no
notes
llama.cpp on MI250 is 3.2x SLOWER than vLLM at identical model/batch (44.4 vs 139.9 t/s) — the paper attributes this to llama.cpp's weak GQA/parallelism exploitation on AMD. Useful as a lower bound, not a MI250 capability ceiling.
Notes
llama.cpp on MI250 is 3.2x SLOWER than vLLM at identical model/batch (44.4 vs 139.9 t/s) — the paper attributes this to llama.cpp's weak GQA/parallelism exploitation on AMD. Useful as a lower bound, not a MI250 capability ceiling.