RTX PRO 2000 Blackwell vLLM: meta-llama/Llama-3.1-8B-Instruct median TTFT, 1000 requests
9976.3 ms (median time to first token, 1000 concurrent)
Fields
id
benchmarks/dm-pro2000-vllm-llama-3-1-8b-c1000-ttftMetric
ttft_ms
Value
9,976.3
Unit
ms (median time to first token, 1000 concurrent)
Model
llama-3-1-8b
Methodology
Database Mart vLLM bench on one RTX PRO 2000 Blackwell 16 GB GDDR7 ECC, 256 GB/s, 70 W. Model meta-llama/Llama-3.1-8B-Instruct, quantization column '16' (bf16/fp16 not distinguished; format_id null, no fp16 record exists). vLLM commit not printed. Single GPU, tensor parallel 1. 1000 requests, input length 100 tokens, output length 600 tokens, benchmark duration 531.73 s, total input tokens 100000, total generated tokens 501950. AGGREGATE across the concurrent requests, not a per-request decode rate.
Measured by
third_party
Reproducible
yes
notes
Median TTFT under the same 1000-request run. P99 TTFT is several times the median on this 16 GB card and is not recorded.
Notes
Median TTFT under the same 1000-request run. P99 TTFT is several times the median on this 16 GB card and is not recorded.