vLLM on 4x MI250: Meta-Llama-3-8B FP16 b1 128in/128out throughput
171.2 tok/s (per-request b1)
Fields
id
benchmarks/argonne-mi250-vllm-metallam-fp16-4x-b1-128io-tpsMetric
tps_aggregate
Value
171.2
Unit
tok/s (per-request b1)
Model
Meta-Llama-3-8B
Methodology
Argonne LLM-Inference-Bench (arXiv:2411.00136), 4x MI250 on Polaris cluster. vLLM 0.6.1 FP16, batch 1, 128in/128out. Throughput = batch*(input+output)/end-to-end latency (Eq.2). per-request batch-1. Reported 'Throughput(tok/s)' column.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.
Notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.