NVIDIA L4, SGLang 0.4.6: Qwen2.5-7B-Instruct FP16 short(256/128) c=64 aggregate tok/s
913.9 output tok/s AGGREGATE across 64 concurrent requests (~14.3 t/s per request)
Fields
id
benchmarks/l4-sglang-qwen25-7b-fp16-short-c64-tpsMetric
tps_aggregate
Value
913.9
Unit
output tok/s AGGREGATE across 64 concurrent requests (~14.3 t/s per request)
Model
Qwen2.5-7B-Instruct (FP16)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: SGLang v0.4.6 (--max-running-req 64 --mem-frac 0.85 --disable-cuda-graph). Model: Qwen2.5-7B-Instruct FP16. Regime 'short' = 256-in / 128-out, concurrency 64, greedy (temp=0). TTFT p50 313 ms at c=64 — SGLang leads throughput at every concurrency level in the sweep. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 64 concurrent requests; divide by 64 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
SGLang FP16 913.9 vs vLLM FP16 831.8 aggregate at c=64 short — +10% for SGLang, despite --disable-cuda-graph. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).
Notes
SGLang FP16 913.9 vs vLLM FP16 831.8 aggregate at c=64 short — +10% for SGLang, despite --disable-cuda-graph. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).