NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-AWQ short(256/128) c=64 aggregate tok/s
975.64 output tok/s AGGREGATE across 64 concurrent requests (~15.2 t/s per request)
Fields
id
benchmarks/l4-vllm-awq-qwen25-7b-short-c64-tpsMetric
tps_aggregate
Value
975.64
Unit
output tok/s AGGREGATE across 64 concurrent requests (~15.2 t/s per request)
Model
Qwen2.5-7B-Instruct-AWQ (INT4)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: vLLM v0.8.5 --quantization awq --enforce-eager. Model: Qwen2.5-7B-Instruct-AWQ. Regime 'short' = 256-in / 128-out, concurrency 64, greedy (temp=0). TTFT p50 651 ms at c=64. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 64 concurrent requests; divide by 64 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
Best measured L4 figure in the sweep: AWQ beats FP16 on aggregate because the smaller weight footprint leaves more KV-cache room.
Notes
Best measured L4 figure in the sweep: AWQ beats FP16 on aggregate because the smaller weight footprint leaves more KV-cache room.