RTX A6000 vLLM: meta-llama/Llama-3.1-8B-Instruct aggregate output throughput, 50 concurrent
1218.63 tok/s (aggregate output tokens, 50 concurrent)
Fields
id
benchmarks/dm-a6000-vllm-llama-3-1-8b-c50-outMetric
tps_aggregate
Value
1,218.63
Unit
tok/s (aggregate output tokens, 50 concurrent)
Model
llama-3-1-8b
Methodology
Database Mart vLLM bench on one RTX A6000 48 GB GDDR6 (page mislabels it 'Quadro RTX A6000'; the Ampere 8.6 / 10752-core spec block is the RTX A6000). Model meta-llama/Llama-3.1-8B-Instruct, quantization column '16' (bf16/fp16 not distinguished; format_id null, no fp16 record exists). vLLM commit not printed. Single GPU, tensor parallel 1. 50 requests, input length 100 tokens, output length 600 tokens, benchmark duration 18.26 s, total input tokens 5000, total generated tokens 22247. AGGREGATE across the concurrent requests, not a per-request decode rate.
Measured by
third_party
Reproducible
yes
notes
50-concurrent scenario only. The page's 100-concurrent table contradicts itself and is not recorded.
Notes
50-concurrent scenario only. The page's 100-concurrent table contradicts itself and is not recorded.