vLLM on 4x MI250: Llama-2-70b-hf FP16 b1 128in/128out throughput
47.7 tok/s (per-request b1)
Fields
id
benchmarks/argonne-mi250-vllm-llama270-fp16-4x-b1-128io-tpsMetric
tps_aggregate
Value
47.7
Unit
tok/s (per-request b1)
Model
Llama-2-70b-hf
Methodology
Argonne LLM-Inference-Bench (arXiv:2411.00136), 4x MI250 on Polaris cluster. vLLM 0.6.1 FP16, batch 1, 128in/128out. Throughput = batch*(input+output)/end-to-end latency (Eq.2). per-request batch-1. Reported 'Throughput(tok/s)' column.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.
Notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.