A40 vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 100 concurrent
1705.49 tok/s (aggregate output tokens, 100 concurrent)
Fields
id
benchmarks/dm-a40-vllm-llama-3-1-8b-c100-outMetric
tps_aggregate
Value
1,705.49
Unit
tok/s (aggregate output tokens, 100 concurrent)
Model
llama-3-1-8b
Methodology
Database Mart vLLM bench on one NVIDIA A40 48 GB GDDR6. Model meta-llama/Llama-3.1-8B-Instruct, quantization column '16' (bf16/fp16 not distinguished; format_id null, no fp16 record exists). vLLM commit not printed. Single GPU, tensor parallel 1. 100 requests, input length 100 tokens, output length 600 tokens, benchmark duration 26.61 s, total input tokens 10000, total generated tokens 45375. AGGREGATE across the concurrent requests, not a per-request decode rate.
Measured by
third_party
Reproducible
yes
notes
Both concurrency levels are internally consistent on this page. 14B rows required --max-model-len 4096 to load.
Notes
Both concurrency levels are internally consistent on this page. 14B rows required --max-model-len 4096 to load.