inference/research
verifiedmedium · 0.80updated 2026-10-03

FasterTransformer

cuda

Fields

id
engines/fastertransformer
Repo
NVIDIA/FasterTransformer
Languages
C++CUDAPython
License
Apache-2.0
First release
2019
Design goal
throughput-first
Scheduling
static-batchingdynamic-batching
Backends
cuda
Formats
fp16int8awqsmoothquantgptq
Notable features
Static and dynamic batching backendsCustom multi-GPU / multi-node kernels with fused GEMM + softmax + samplingVendor fork with INT8 quantization paths; was the default reference serving stack before vLLM
Hardware caveats
CUDA-only in practiceSuccessor path is TensorRT-LLM; FasterTransformer is now maintenance-mode at NVIDIA
Best for
offline / static-batch throughput on NVIDIA GPUs
Avoid for
low-latency interactive serving - no continuous batching
notes
The 'TensorRT-LLM-based inference server' in the Databricks best-practices post, and the baseline engine in the vLLM PagedAttention paper and in the SmoothQuant paper (which integrates SmoothQuant into FasterTransformer for its INT8 speed claims). Recorded so benchmark records can point at an engine id instead of free-text engine names. SmoothQuant's 1.56x speedup over FP16 is a FasterTransformer INT8 result, NOT a vLLM or TensorRT-LLM result. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2023-10-19 (35.5 months before this record), latest release release/v5.3_tag published 2023-01-23, 40 contributors (distinct committers including anonymous), open_issues_count 288 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 6457 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.

Notes

The 'TensorRT-LLM-based inference server' in the Databricks best-practices post, and the baseline engine in the vLLM PagedAttention paper and in the SmoothQuant paper (which integrates SmoothQuant into FasterTransformer for its INT8 speed claims). Recorded so benchmark records can point at an engine id instead of free-text engine names. SmoothQuant's 1.56x speedup over FP16 is a FasterTransformer INT8 result, NOT a vLLM or TensorRT-LLM result. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2023-10-19 (35.5 months before this record), latest release release/v5.3_tag published 2023-01-23, 40 contributors (distinct committers including anonymous), open_issues_count 288 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 6457 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.

Cross-references

This record references no other records — it is a leaf. It is cited by 7.