RTX A4000 Ollama 0.5.11: llama3.1:8b per-request eval rate
51.35 tok/s (per-request decode, batch size 1)
Fields
id
benchmarks/dm-a4000-ollama-llama-31-8b-evalMetric
decode_tok_s
Value
51.35
Unit
tok/s (per-request decode, batch size 1)
Model
llama-3.1-8b
Methodology
Database Mart Ollama 0.5.11 on one RTX A4000 16 GB GDDR6 (Windows VPS, 24 cores, 32 GB RAM). Default Ollama pull, labelled 4-bit, exact GGUF quant not printed, so format_id is null. Prompt is a single short question and the output length is model-determined, so this is the eval-rate column only: per-request decode, batch size 1, NOT aggregate and NOT a prefill rate. The llama.cpp commit inside Ollama 0.5.11 is not printed. Ollama tag llama3.1:8b. Reported GPU utilization 78%.
Measured by
third_party
Reproducible
yes
notes
Eval Rate (tokens/s) column. Per-request, not aggregate.
Notes
Eval Rate (tokens/s) column. Per-request, not aggregate.