inference/research
verifiedmedium · 0.80updated 2026-10-03

SmoothQuant reference implementation (PyTorch W8A8)

none

Fields

id
engines/smoothquant-reference
Repo
mit-han-lab/smoothquant
Languages
Python
License
MIT
First release
2022
Design goal
minimal-deps
Scheduling
none
Backends
none
Formats
int8smoothquant
Notable features
Offline outlier migration then INT8 linear layersW8A8 via torch._int_mm / fake-quant W8A8W8 pathThree smoothing strengths O1 (per-channel static) / O2 (per-token dynamic) / O3 (per-tensor static)
Hardware caveats
The reference repo is an evaluation harness, not a serving runtime: its INT8 path is PyTorch, so its own latency numbers are not representative of the FasterTransformer INT8 kernels used for the paper's speed claims
Best for
reproducing the paper's accuracy numbers
Avoid for
latency measurement
notes
IMPORTANT methodology point for every SmoothQuant quality delta in this repo: the accuracy numbers come from this PyTorch harness (evaluated with lm-eval-harness), while the 1.51x/1.56x speedups come from the FasterTransformer INT8 integration. Accuracy and speed in that paper are NOT from the same runtime. O1-O3 are different schemes, not different strengths of one scheme: O3 is per-tensor static and is the one that loses accuracy on OPT-175B. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2024-07-12 (26.7 months before this record), NO GitHub release or tag found, 4 contributors (distinct committers including anonymous), open_issues_count 72 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 1692 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.

Notes

IMPORTANT methodology point for every SmoothQuant quality delta in this repo: the accuracy numbers come from this PyTorch harness (evaluated with lm-eval-harness), while the 1.51x/1.56x speedups come from the FasterTransformer INT8 integration. Accuracy and speed in that paper are NOT from the same runtime. O1-O3 are different schemes, not different strengths of one scheme: O3 is per-tensor static and is the one that loses accuracy on OPT-175B. ||| MAINTENANCE (dated fact, GitHub API, 2026-10-04): last commit 2024-07-12 (26.7 months before this record), NO GitHub release or tag found, 4 contributors (distinct committers including anonymous), open_issues_count 72 (issues PLUS pull requests - see [[ev-github-open-issues-count-includes-pull-requests]]), not archived, 1692 stars. See [[ev-maintenance-status-is-four-dated-facts]] for how to read this fact set. ||| DEPLOYMENT: NO EVIDENCE OF PRODUCTION DEPLOYMENT FOUND as of 2026-10-04 - repositories, READMEs, docs and issue trackers were searched. See [[ev-production-evidence-is-scarce-and-nearly-always-single-site]] for the grading scheme; note that for an artifact-evaluation or paper-code release, non-deployment is documented intent rather than a negative finding.

Cross-references

This record references no other records — it is a leaf. It is cited by 4.