inference/research
contestedmedium · 0.70updated 2026-10-03

SmoothQuant W8A8 7-task average accuracy delta vs fp16, BLOOM-176B

-1.6 accuracy_delta_pp (average zero-shot accuracy percentage points, quantized minus FP16)

Fields

id
benchmarks/smoothquant-w8a8-zeroshot-accuracy-delta-bloom-176b
Metric
quality
Value
-1.6
Unit
accuracy_delta_pp (average zero-shot accuracy percentage points, quantized minus FP16)
Model
BLOOM-176B
Methodology
SmoothQuant paper Table 4. Model BLOOM-176B, W8A8, scheme O3 (per-tensor static). Metric: unweighted average accuracy over WinoGrande, HellaSwag, PIQA and LAMBADA - the paper explicitly states this column is NOT column-wise comparable across the three models because OPT and BLOOM are scored on different dataset mixes. FP16 baseline 68.2 percent; SmoothQuant-O3 scores 67.4 percent; O2 68.4; O1 68.3.
Measured by
third_party
Reproducible
yes
notes
DELTA = -1.6 pp (67.4 vs 68.2) for O3. A 'SmoothQuant costs BLOOM-176B 1.6 points' claim is therefore scheme-specific and is contradicted by the paper's own adjacent rows: O1 scores 68.3 and O2 scores 68.4, both ABOVE the 68.2 fp16 baseline. Hence contested, not resolved. The paper attributes the O3 deficit to a discrepancy between statically collected activation statistics and real evaluation samples, which is a hypothesis rather than a measurement. Second point: BLOOM-176B is the EASY quantization case. The same table shows naive W8A8 per-tensor dynamic quantization costs only about 4 points here (64.2 vs 68.2) versus 39.3 points on OPT-175B (32.3 vs 71.6). Quantization difficulty is a property of the model, not of the format, so a BLOOM-176B delta is not evidence about OPT.

Notes

DELTA = -1.6 pp (67.4 vs 68.2) for O3. A 'SmoothQuant costs BLOOM-176B 1.6 points' claim is therefore scheme-specific and is contradicted by the paper's own adjacent rows: O1 scores 68.3 and O2 scores 68.4, both ABOVE the 68.2 fp16 baseline. Hence contested, not resolved. The paper attributes the O3 deficit to a discrepancy between statically collected activation statistics and real evaluation samples, which is a hypothesis rather than a measurement. Second point: BLOOM-176B is the EASY quantization case. The same table shows naive W8A8 per-tensor dynamic quantization costs only about 4 points here (64.2 vs 68.2) versus 39.3 points on OPT-175B (32.3 vs 71.6). Quantization difficulty is a property of the model, not of the format, so a BLOOM-176B delta is not evidence about OPT.

Cross-references

Engine (1)

Format (1)