NVIDIA L4, vLLM 0.8.5: Qwen3-8B-AWQ reasoning(256/~8192) c=16 aggregate tok/s
345.49 output tok/s AGGREGATE across 16 concurrent reasoning requests (~21.6 t/s per request)
Fields
id
benchmarks/l4-vllm-awq-qwen3-8b-reasoning-c16-tpsMetric
tps_aggregate
Value
345.49
Unit
output tok/s AGGREGATE across 16 concurrent reasoning requests (~21.6 t/s per request)
Model
Qwen3-8B-AWQ (INT4, thinking tokens)
Methodology
THIRD-PARTY MEASURED (ree2raz/inference-bench, Modal L4 24GB, client+server collocated). Engine: vLLM v0.8.5 --quantization awq --reasoning-parser deepseek_r1 --max-model-len 16384. Model: Qwen3-8B-AWQ. Regime 'reasoning' = ~256-in / ~8192-out, concurrency 16, greedy (temp=0). TTFT->answer p50 266 s at c=16. Values are medians verbatim from results/summary.csv. throughput_median is AGGREGATE output tok/s across all 16 concurrent requests; divide by 16 for the per-request rate.
Measured by
third_party
Reproducible
yes
notes
2.0x the FP16 aggregate on reasoning because AWQ KV-cache pressure is lower — all 16 requests run concurrently instead of queuing.
Notes
2.0x the FP16 aggregate on reasoning because AWQ KV-cache pressure is lower — all 16 requests run concurrently instead of queuing.