NVIDIA L4, SGLang 0.4.6: Qwen2.5-7B-Instruct FP16 short(256/128) c=1 output tok/s
17.12 output tok/s at concurrency 1 (= per-request ~17.1 t/s)
Fields
id
benchmarks/l4-sglang-qwen25-7b-fp16-short-c1-tpsMetric
tps_aggregate
Value
17.12
Unit
output tok/s at concurrency 1 (= per-request ~17.1 t/s)
Model
Qwen2.5-7B-Instruct (FP16)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: SGLang v0.4.6 --disable-cuda-graph. Model: Qwen2.5-7B-Instruct FP16. Regime 'short' = 256-in / 128-out, concurrency 1, greedy (temp=0). TTFT p50 118.6 ms. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 1 concurrent requests; divide by 1 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
c=1 SGLang FP16 ≈ vLLM FP16 (~17 t/s) — the engine doesn't matter for single-user decode on a 300 GB/s card, the model read does. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).
Notes
c=1 SGLang FP16 ≈ vLLM FP16 (~17 t/s) — the engine doesn't matter for single-user decode on a 300 GB/s card, the model read does. format_id=null: no fp16 record exists in data/quantization/ (FP16 is the unquantized baseline, not a format).