vLLM 0.1.4 (EmbeddedLLM ROCm port) on amd-instinct-mi210: LLaMA2-13B 2x MI210 TP=2 offline throughput tok/s
595.11 tok/s aggregate (offline inference, 1000 questions; NOT per-request)
Fields
id
benchmarks/embeddedllm-vllm-mi210-llama2-13b-2g-tpsMetric
tps_aggregate
Value
595.11
Unit
tok/s aggregate (offline inference, 1000 questions; NOT per-request)
Model
LLaMA2 (HF bf16, unquantized)
Methodology
THIRD-PARTY MEASURED (EmbeddedLLM blog 'vllm_rocm', Oct 27 2023). LLaMA2-13B 2x MI210 TP=2. vLLM v0.1.4 EmbeddedLLM ROCm port for MI210 (vllm-rocm repo); offline throughput benchmark over 1000 questions - aggregate tokens/sec for the whole run. Requests/min companion column corroborates ordering. Batch is the full 1000-question sweep, not a fixed concurrency.
Measured by
third_party
Reproducible
yes
notes
Early vLLM 0.1.4 era - pre-continuous-batching maturity, so a LOWER bound vs modern vLLM. MI210 parity claim is vs A100-80GB on identical workload.
Notes
Early vLLM 0.1.4 era - pre-continuous-batching maturity, so a LOWER bound vs modern vLLM. MI210 parity claim is vs A100-80GB on identical workload.