NVIDIA L4, llama.cpp b5540 CUDA: Qwen2.5-7B Q4_K_M short(256/128) c=1 output tok/s
47.37 output tok/s at concurrency 1 (= per-request ~47.4 t/s)
Fields
id
benchmarks/l4-llamacpp-qwen25-7b-q4km-short-c1-tpsMetric
tps_aggregate
Value
47.37
Unit
output tok/s at concurrency 1 (= per-request ~47.4 t/s)
Model
Qwen2.5-7B-Instruct-GGUF Q4_K_M (~4.4 GB)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: llama.cpp b5540 (-np 4 --parallel 4 -ngl 99 -c 16384). Model: Qwen2.5-7B-Instruct-GGUF Q4_K_M. Regime 'short' = 256-in / 128-out, concurrency 1, greedy (temp=0). llama.cpp server, no streaming metrics (no TTFT/TPOT). Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 1 concurrent requests; divide by 1 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
llama.cpp is FASTEST single-user engine on the L4: 47.4 t/s at c=1 vs ~17 t/s vLLM/SGLang FP16 (Q4_K_M weights are 4.4 GB vs 15 GB FP16 — mostly a quant effect, not an engine win).
Notes
llama.cpp is FASTEST single-user engine on the L4: 47.4 t/s at c=1 vs ~17 t/s vLLM/SGLang FP16 (Q4_K_M weights are 4.4 GB vs 15 GB FP16 — mostly a quant effect, not an engine win).