NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct FP16 short(256/128) c=1 output tok/s
17.11 output tok/s at concurrency 1 (= per-request decode ~17.1 t/s)
Fields
id
benchmarks/l4-vllm-qwen25-7b-fp16-short-c1-tpsMetric
tps_aggregate
Value
17.11
Unit
output tok/s at concurrency 1 (= per-request decode ~17.1 t/s)
Model
Qwen2.5-7B-Instruct (FP16)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: vLLM v0.8.5 (--max-num-seqs 64 --gpu-mem-util 0.90). Model: Qwen2.5-7B-Instruct FP16 (~15 GB). Regime 'short' = 256-in / 128-out, concurrency 1, greedy (temp=0). c=1 is the single-request case, so aggregate = per-request. TTFT p50 69.4 ms, TPOT p50 58.3 ms. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 1 concurrent requests; divide by 1 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
The L4 single-user datapoint: ~17 tok/s decode on a 7B FP16. TTFT p50 0.0694 s is the lowest c=1 TTFT of the three engines. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).
Notes
The L4 single-user datapoint: ~17 tok/s decode on a 7B FP16. TTFT p50 0.0694 s is the lowest c=1 TTFT of the three engines. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).