vLLM 0.1.4 (upstream CUDA) on nvidia-l40s: LLaMA2-13B 2x L40 TP=2 offline throughput tok/s
491.16 tok/s aggregate (offline inference, 1000 questions; NOT per-request)
Fields
id
benchmarks/embeddedllm-vllm-l40-llama2-13b-2g-tpsMetric
tps_aggregate
Value
491.16
Unit
tok/s aggregate (offline inference, 1000 questions; NOT per-request)
Model
LLaMA2 (HF bf16, unquantized)
Methodology
THIRD-PARTY MEASURED (EmbeddedLLM blog 'vllm_rocm', Oct 27 2023). LLaMA2-13B 2x L40 TP=2. vLLM v0.1.4 upstream CUDA build; offline throughput benchmark over 1000 questions - aggregate tokens/sec for the whole run. Requests/min companion column corroborates ordering. Batch is the full 1000-question sweep, not a fixed concurrency.
Measured by
third_party
Reproducible
yes
notes
Early vLLM 0.1.4 era - pre-continuous-batching maturity, so a LOWER bound vs modern vLLM. MI210 parity claim is vs A100-80GB on identical workload.
Notes
Early vLLM 0.1.4 era - pre-continuous-batching maturity, so a LOWER bound vs modern vLLM. MI210 parity claim is vs A100-80GB on identical workload.