NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct-AWQ short(256/128) c=1 output tok/s
43.19 output tok/s at concurrency 1 (= per-request ~43.2 t/s)
Fields
id
benchmarks/l4-vllm-awq-qwen25-7b-short-c1-tpsMetric
tps_aggregate
Value
43.19
Unit
output tok/s at concurrency 1 (= per-request ~43.2 t/s)
Model
Qwen2.5-7B-Instruct-AWQ (INT4)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: vLLM v0.8.5 --quantization awq --enforce-eager. Model: Qwen2.5-7B-Instruct-AWQ (~5.2 GB). Regime 'short' = 256-in / 128-out, concurrency 1, greedy (temp=0). TTFT p50 35.6 ms, TPOT p50 22.9 ms — 2.5x the FP16 single-user rate and 1/3 the VRAM. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 1 concurrent requests; divide by 1 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
AWQ on L4 is the 'free lunch' the author calls out: 43 t/s single-user vs 17 t/s FP16, and 975 t/s aggregate at c=64 — higher than FP16 because more KV cache fits.
Notes
AWQ on L4 is the 'free lunch' the author calls out: 43 t/s single-user vs 17 t/s FP16, and 975 t/s aggregate at c=64 — higher than FP16 because more KV cache fits.