NVIDIA L4, llama.cpp b5540 CUDA: Qwen2.5-7B Q4_K_M short(256/128) c=64 aggregate tok/s
189.19 output tok/s AGGREGATE across 64 concurrent requests (~2.96 t/s per request)
Fields
id
benchmarks/l4-llamacpp-qwen25-7b-q4km-short-c64-tpsMetric
tps_aggregate
Value
189.19
Unit
output tok/s AGGREGATE across 64 concurrent requests (~2.96 t/s per request)
Model
Qwen2.5-7B-Instruct-GGUF Q4_K_M
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: llama.cpp b5540 (-np 4 --parallel 4 -ngl 99 -c 16384). Model: Qwen2.5-7B-Instruct-GGUF Q4_K_M. Regime 'short' = 256-in / 128-out, concurrency 64, greedy (temp=0). Latency p95 73.4 s at c=64 — llama.cpp scales poorly past c=4. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 64 concurrent requests; divide by 64 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
The serving cliff: 189 t/s aggregate at c=64 vs 914 (SGLang) / 832 (vLLM) — llama.cpp is the single-user winner, not the server winner.
Notes
The serving cliff: 189 t/s aggregate at c=64 vs 914 (SGLang) / 832 (vLLM) — llama.cpp is the single-user winner, not the server winner.