SmoothQuant W8A8 WikiText perplexity delta vs fp16, OPT-175B

Fields

id

benchmarks/smoothquant-w8a8-wikitext-ppl-delta-opt-175b

Metric

quality

Value

0.18

Unit

perplexity_delta (WikiText ppl points, quantized minus FP16)

Model

OPT-175B

Methodology

SmoothQuant paper Table 3. Model OPT-175B, TRUE W8A8: 8-bit weights times 8-bit activations with int32 accumulate, scheme O3 (the most aggressive of the three: per-tensor STATIC activation scales). Static scales calibrated offline from sampled calibration sentences. Metric: WikiText perplexity. FP16 baseline 10.99; SmoothQuant-O3 scores 11.17. Same table: O1 (per-channel static) 11.11, O2 (per-token dynamic) 11.14, LLM.int8() 11.10.

Measured by

third_party

Reproducible

yes

notes

DELTA = +0.18 ppl (11.17 vs 10.99) for scheme O3. The misreading this record exists to prevent: in the same table, naive W8A8 per-tensor quantization of the same model scores 93080 - a four-to-five order of magnitude perplexity blowup that renders the model useless. ZeroQuant is 84648, Outlier Suppression 96151. So 'W8A8 costs 0.18 perplexity points' is true ONLY of the smoothed model, and the unsmoothed version of the identical precision costs 90000x more. Second trap: this number is meaningless without saying which of O1/O2/O3, because the three schemes differ by 0.06 ppl here and by up to 1.3 accuracy points on OPT-175B elsewhere. Third: the paper's 1.51x/1.56x speedups come from its FasterTransformer INT8 integration, a completely different runtime from the PyTorch harness that produced these accuracy numbers - accuracy and speed in that paper are not from the same engine.

Notes

DELTA = +0.18 ppl (11.17 vs 10.99) for scheme O3. The misreading this record exists to prevent: in the same table, naive W8A8 per-tensor quantization of the same model scores 93080 - a four-to-five order of magnitude perplexity blowup that renders the model useless. ZeroQuant is 84648, Outlier Suppression 96151. So 'W8A8 costs 0.18 perplexity points' is true ONLY of the smoothed model, and the unsmoothed version of the identical precision costs 90000x more. Second trap: this number is meaningless without saying which of O1/O2/O3, because the three schemes differ by 0.06 ppl here and by up to 1.3 accuracy points on OPT-175B elsewhere. Third: the paper's 1.51x/1.56x speedups come from its FasterTransformer INT8 integration, a completely different runtime from the PyTorch harness that produced these accuracy numbers - accuracy and speed in that paper are not from the same engine.

SmoothQuant W8A8 WikiText perplexity delta vs fp16, OPT-175B

Fields

Notes

Cross-references

Engine (1)

Format (1)