NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct FP16 short(256/128) c=64 aggregate tok/s
831.76 output tok/s AGGREGATE across 64 concurrent requests (~13.0 t/s per request)
Fields
id
benchmarks/l4-vllm-qwen25-7b-fp16-short-c64-tpsMetric
tps_aggregate
Value
831.76
Unit
output tok/s AGGREGATE across 64 concurrent requests (~13.0 t/s per request)
Model
Qwen2.5-7B-Instruct (FP16)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: vLLM v0.8.5 (--max-num-seqs 64 --gpu-mem-util 0.90). Model: Qwen2.5-7B-Instruct FP16. Regime 'short' = 256-in / 128-out, concurrency 64, greedy (temp=0). TTFT p50 578 ms, TPOT p50 75.7 ms at c=64. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 64 concurrent requests; divide by 64 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
Saturated-serving figure; the per-request rate (~13 t/s) is nearly flat from c=1 to c=64 which is the interesting finding — L4 FP16 KV-cache pressure queues requests rather than slowing decode. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).
Notes
Saturated-serving figure; the per-request rate (~13 t/s) is nearly flat from c=1 to c=64 which is the interesting finding — L4 FP16 KV-cache pressure queues requests rather than slowing decode. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).