ree2raz/inference-bench — reproducible head-to-head LLM serving benchmark on a single NVIDIA L4 (vLLM v0.8.5 / SGLang v0.4.6 / llama.cpp b5540, results/summary.csv)

Single NVIDIA L4 24GB on Modal (client+server collocated in the same container, greedy temp=0). Models: Qwen2.5-7B-Instruct FP16 + AWQ (vllm/sglang), Qwen2.5-7B-Instruct-GGUF Q4_K_M (llama.cpp -np 4 -ngl 99 -c 16384), Qwen3-8B FP16 + AWQ reasoning. Regimes: short 256in/128out, long 2048in/512out, reasoning ~256in/8192out. summary.csv rows are medians over 3 repeats for short (1 repeat for long/reasoning). throughput_median = AGGREGATE output tok/s across the concurrency level; throughput_per_request_median is the per-request rate. ttft_p50_median seconds. Raw JSONL in Modal volume; committed CSV is the deduplicated summary.