llama.cpp on MI250: Llama-2-7b FP16 b1 128io throughput
44.4 tok/s (per-request b1)
Fields
id
benchmarks/argonne-mi250-llamacpp-l27b-fp16-1x-b1-128io-tpsMetric
tps_aggregate
Value
44.4
Unit
tok/s (per-request b1)
Model
Llama-2-7b-hf
Methodology
Argonne LLM-Inference-Bench (arXiv:2411.00136), 1x MI250 on Polaris cluster. llama.cpp FP16, batch 1, 128in/128out. Throughput = batch*(input+output)/end-to-end latency (Eq.2). per-request batch-1. Reported 'Throughput(tok/s)' column.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
llama.cpp HIP/ROCm path is far below vLLM on MI250 - kept as lower bound.
Notes
llama.cpp HIP/ROCm path is far below vLLM on MI250 - kept as lower bound.