GuideLLM (Red Hat): A100-PCIE-40GB, vLLM 0.9.1, Llama-3.1-8B fp16, peak aggregate output tok/s
793.0 output tok/s (aggregate at saturation, concurrency up to 256)
Fields
id
benchmarks/guidellm-a100-pcie-40gb-llama31-8b-fp16-vllm-peak-tpsMetric
tps_aggregate
Value
793
Unit
output tok/s (aggregate at saturation, concurrency up to 256)
Model
meta-llama/Llama-3.1-8B-Instruct
Methodology
THIRD-PARTY LAB MEASURED (Red Hat GuideLLM v0.2.1 study, Aug 2025; republished by Markaicode). Single NVIDIA A100-PCIE-40GB, driver 550.144.03, CUDA 12.4, OpenShift 4.17.15. vLLM 0.9.1 serving meta-llama/Llama-3.1-8B-Instruct FP16. Concurrency ramped 1..256, 300 s per level, fixed prompt-response dataset. Reported peak output TPS at saturation; P99 TTFT at peak = 80 ms. AGGREGATE across concurrent requests, NOT per-request decode.
Measured by
third_party
Reproducible
yes
notes
The aggregate number for the part. vLLM's continuous batching is what produces the 793 figure — per-request decode on the same card is in the ~30-45 tok/s band. Companion Ollama record shows the contrast. This is the only PCIe-40GB measurement the repo has; the two A100-80GB-SXM4 records are different silicon clocks (400 W vs 250 W cap).
Notes
The aggregate number for the part. vLLM's continuous batching is what produces the 793 figure — per-request decode on the same card is in the ~30-45 tok/s band. Companion Ollama record shows the contrast. This is the only PCIe-40GB measurement the repo has; the two A100-80GB-SXM4 records are different silicon clocks (400 W vs 250 W cap).