vLLM on 4x MI250: Meta-Llama-3-8B FP16 b64 128in/128out throughput
5922.6 tok/s (aggregate x64)
Fields
id
benchmarks/argonne-mi250-vllm-metallam-fp16-4x-b64-128io-tpsMetric
tps_aggregate
Value
5,922.6
Unit
tok/s (aggregate x64)
Model
Meta-Llama-3-8B
Methodology
Argonne LLM-Inference-Bench (arXiv:2411.00136), 4x MI250 on Polaris cluster. vLLM 0.6.1 FP16, batch 64, 128in/128out. Throughput = batch*(input+output)/end-to-end latency (Eq.2). AGGREGATE across batch. Reported 'Throughput(tok/s)' column.
Measured by
third_party
Reproducible
yes
Thermal state
unknown
notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.
Notes
MI250 (128GB HBM2e, 3.28 TB/s) on Argonne Polaris. First measured record on amd-instinct-mi250.