vLLM on 1x MI250: Llama-2-7b-hf FP16 b64 128in/128out throughput
4477.1 tok/s (aggregate x64)
Fields
id
benchmarks/argonne-mi250-vllm-llama27b-fp16-1x-b64-128io-tpsMetric
tps_aggregate
Value
4,477.1
Unit
tok/s (aggregate x64)
Model
Llama-2-7b-hf
Methodology
Argonne LLM-Inference-Bench (arXiv:2411.00136), 1x MI250 on Polaris cluster. vLLM 0.6.1 FP16, batch 64, 128in/128out. Throughput = batch*(input+output)/end-to-end latency (Eq.2). AGGREGATE across batch. Reported 'Throughput(tok/s)' column.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.
Notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.