Embedded LLM - High throughput LLM inference with vLLM and AMD (MI210 vs A100 vs L40)

Fields

id

sources/embeddedllm-vllm-rocm-mi210-vs-a100-l40-throughput

URL

https://embeddedllm.com/blog/vllm_rocm

Publisher

Embedded LLM

Kind

benchmark

Accessed

2026-10-14

notes

Oct 27 2023. vLLM v0.1.4 (EmbeddedLLM ROCm port) on MI210 64GB HBM2e; vLLM v0.1.4 on A100-80GB-SXM and L40. Offline throughput of 1000 questions, LLaMA2-7B (1 GPU), 13B (2 GPU TP), 70B (4 GPU TP). Tables 3+4: tok/s and req/min. Third-party measured.

Notes

Oct 27 2023. vLLM v0.1.4 (EmbeddedLLM ROCm port) on MI210 64GB HBM2e; vLLM v0.1.4 on A100-80GB-SXM and L40. Offline throughput of 1000 questions, LLaMA2-7B (1 GPU), 13B (2 GPU TP), 70B (4 GPU TP). Tables 3+4: tok/s and req/min. Third-party measured.

Cross-references

This record references no other records.