GuideLLM (Red Hat): A100-PCIE-40GB, Ollama 0.9.2 (NUM_PARALLEL=32), Llama-3.1-8B fp16, peak aggregate output tok/s
41.0 output tok/s (aggregate at saturation, concurrency up to 256)
Fields
id
benchmarks/guidellm-a100-pcie-40gb-llama31-8b-fp16-ollama-peak-tpsMetric
tps_aggregate
Value
41
Unit
output tok/s (aggregate at saturation, concurrency up to 256)
Model
meta-llama/Llama-3.1-8B-Instruct
Methodology
THIRD-PARTY LAB MEASURED (Red Hat GuideLLM v0.2.1 study, Aug 2025; republished by Markaicode). Same host as the vLLM row: single A100-PCIE-40GB, CUDA 12.4, OpenShift 4.17.15. Ollama 0.9.2 with OLLAMA_NUM_PARALLEL=32 (the highest stable value on this card). llama3.1:8b-instruct-fp16. Peak output TPS at saturation; P99 TTFT at peak = 673 ms. AGGREGATE.
Measured by
third_party
Reproducible
yes
notes
Same-card, same-model, same-harness counterpart to the vLLM 793 tok/s record — the ~19x gap is a scheduler (continuous batching vs bounded parallel slots) story, not a hardware capability difference. Do NOT read 41 tok/s as the card's decode ceiling.
Notes
Same-card, same-model, same-harness counterpart to the vLLM 793 tok/s record — the ~19x gap is a scheduler (continuous batching vs bounded parallel slots) story, not a hardware capability difference. Do NOT read 41 tok/s as the card's decode ceiling.