inference/research
verifiedlow · 0.60updated 2026-10-17

A40 vLLM: deepseek-ai/DeepSeek-R1-Distill-Qwen-14B aggregate output throughput, 50 concurrent

446.73 tok/s (aggregate output tokens, 50 concurrent)

Fields

id
benchmarks/dm-a40-vllm-deepseek-r1-distill-qwen-14b-c50-out
Metric
tps_aggregate
Value
446.73
Unit
tok/s (aggregate output tokens, 50 concurrent)
Model
deepseek-r1-distill-qwen-14b
Methodology
Database Mart hosted-GPU report, page footer dated 10/07/2026. Engine is vLLM built from github.com/vllm-project/vllm with no release tag or commit printed, so the engine version is unpinned. Weights are the Hugging Face checkpoint at the precision the table labels 'Quantization 16' (bf16/fp16 not distinguished); format_id is null because data/quantization/ has no fp16/bf16 record. Single GPU, tensor parallel 1. Input length fixed at 100 tokens, output length fixed at 600 tokens. The 'Output (tokens/s)' column is aggregate generated tokens across the concurrent requests divided by benchmark duration, NOT a per-request decode rate. Median TTFT is the median time to first token under that same concurrency. Third-party measurement by the GPU host; confidence capped because the harness and the vLLM commit are unpublished. Card: NVIDIA A40, Ampere, 48 GB GDDR6, 10,752 CUDA cores. 50 requests, input length 100, output length 600, benchmark duration 26.72 s, total input tokens 5000, total generated tokens 11937. AGGREGATE across the concurrent requests, not a per-request decode rate.
Measured by
third_party
Reproducible
yes
notes
Output (tokens/s) column. These six model rows were not in the earlier A40 filing, which covered Llama 3.1 8B and Qwen2.5 7B/14B only.

Notes

Output (tokens/s) column. These six model rows were not in the earlier A40 filing, which covered Llama 3.1 8B and Qwen2.5 7B/14B only.

Cross-references