NVIDIA L4, vLLM 0.8.5: Qwen2.5-7B-Instruct FP16 long(2048/512) c=64 aggregate tok/s
776.53 output tok/s AGGREGATE across 64 concurrent requests (~12.1 t/s per request)
Fields
id
benchmarks/l4-vllm-qwen25-7b-fp16-long-c64-tpsMetric
tps_aggregate
Value
776.53
Unit
output tok/s AGGREGATE across 64 concurrent requests (~12.1 t/s per request)
Model
Qwen2.5-7B-Instruct (FP16)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: vLLM v0.8.5. Model: Qwen2.5-7B-Instruct FP16. Regime 'long' = 2048-in / 512-out, concurrency 64, greedy (temp=0). TTFT p50 681 ms at c=64 long. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 64 concurrent requests; divide by 64 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
2K-token prompts barely dent aggregate throughput vs the short regime (776 vs 831 t/s) — decode-bandwidth-bound, not prefill-bound. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).
Notes
2K-token prompts barely dent aggregate throughput vs the short regime (776 vs 831 t/s) — decode-bandwidth-bound, not prefill-bound. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).