vLLM 0.1.4 (upstream CUDA) on nvidia-l40s: LLaMA2-70B 4x L40 TP=4 offline throughput tok/s
399.01 tok/s aggregate (offline inference, 1000 questions; NOT per-request)
Fields
id
benchmarks/embeddedllm-vllm-l40-llama2-70b-4g-tpsMetric
tps_aggregate
Value
399.01
Unit
tok/s aggregate (offline inference, 1000 questions; NOT per-request)
Model
LLaMA2 (HF bf16, unquantized)
Methodology
THIRD-PARTY MEASURED (EmbeddedLLM blog 'vllm_rocm', Oct 27 2023). LLaMA2-70B 4x L40 TP=4. vLLM v0.1.4 upstream CUDA build; offline throughput benchmark over 1000 questions - aggregate tokens/sec for the whole run. Requests/min companion column corroborates ordering. Batch is the full 1000-question sweep, not a fixed concurrency.
Measured by
third_party
Reproducible
yes
notes
Early vLLM 0.1.4 era - pre-continuous-batching maturity, so a LOWER bound vs modern vLLM. MI210 parity claim is vs A100-80GB on identical workload.
Notes
Early vLLM 0.1.4 era - pre-continuous-batching maturity, so a LOWER bound vs modern vLLM. MI210 parity claim is vs A100-80GB on identical workload.