inference/research
verifiedlow · 0.60updated 2026-10-17

A100 40GB PCIe vLLM: Qwen/Qwen2.5-14B-Instruct aggregate output throughput, 50 concurrent

420.31 tok/s (aggregate output tokens, 50 concurrent)

Fields

id
benchmarks/dm-a100-40gb-vllm-qwen25-14b-instruct-c50-out
Metric
tps_aggregate
Value
420.31
Unit
tok/s (aggregate output tokens, 50 concurrent)
Model
qwen2.5-14b-instruct
Methodology
Database Mart hosted-GPU report, page footer dated 10/07/2026. Engine is vLLM built from github.com/vllm-project/vllm with no release tag or commit printed, so the engine version is unpinned. Weights are the Hugging Face checkpoint at the precision the table labels 'Quantization 16' (bf16/fp16 not distinguished); format_id is null because data/quantization/ has no fp16/bf16 record. Single GPU, tensor parallel 1. Input length fixed at 100 tokens, output length fixed at 600 tokens. The 'Output (tokens/s)' column is aggregate generated tokens across the concurrent requests divided by benchmark duration, NOT a per-request decode rate. Median TTFT is the median time to first token under that same concurrency. Third-party measurement by the GPU host; confidence capped because the harness and the vLLM commit are unpublished. Card: A100 40 GB PCIe (6,912 CUDA cores, 40 GB HBM2), not the SXM4 or 80 GB cards. 50 requests, input length 100, output length 600, benchmark duration 27.52 s, total input tokens 5000, total generated tokens 11567. AGGREGATE across the concurrent requests, not a per-request decode rate.
Measured by
third_party
Reproducible
yes
notes
Output (tokens/s) column. 14B row required --max-model-len 4096 to load.

Notes

Output (tokens/s) column. 14B row required --max-model-len 4096 to load.

Cross-references