SmoothQuant with FP8 activations: does outlier migration still help?

Fields

id

quantization/qhw-smoothquant-fp8-interaction

Scheme

smoothquant

Bits

w8a8 (8-bit weights, 8-bit activations)

Activation scheme

SmoothQuant migrates quantization difficulty from activations to weights using a per-channel transformation. When the activation is already FP8, the question is whether the migration still helps. The answer depends on whether FP8's limited precision (E4M3 has ~2 decimal digits) is sufficient to represent the migrated scales.

Kernels

notes

GROUP SIZE LEFT NULL 2026-10-04: the subject is whether SmoothQuant's outlier migration still helps when activations are FP8. The grouping is per-tensor on both sides; there is no weight group to record. This is a research question rather than a documented result. The key issue: SmoothQuant's migration transforms activations by dividing by s_j and multiplying weights by s_j. When the activation is quantized to FP8 E4M3, the migrated activation has a different distribution that may or may not be easier to quantize. The FP8 format has limited precision (E4M3 has 3 mantissa bits), so the migration may not help as much as it does for INT8. The existing SmoothQuant record shows it was designed for INT8, not FP8. The interaction between SmoothQuant and FP8 is an open research question. PRACTICAL IMPLICATION: if you are using FP8 activations, you may not need SmoothQuant at all - the FP8 format already handles outliers better than INT8 because the exponent provides dynamic range. But this has not been verified with a controlled benchmark. This is a priority for further research.

Notes

GROUP SIZE LEFT NULL 2026-10-04: the subject is whether SmoothQuant's outlier migration still helps when activations are FP8. The grouping is per-tensor on both sides; there is no weight group to record. This is a research question rather than a documented result. The key issue: SmoothQuant's migration transforms activations by dividing by s_j and multiplying weights by s_j. When the activation is quantized to FP8 E4M3, the migrated activation has a different distribution that may or may not be easier to quantize. The FP8 format has limited precision (E4M3 has 3 mantissa bits), so the migration may not help as much as it does for INT8. The existing SmoothQuant record shows it was designed for INT8, not FP8. The interaction between SmoothQuant and FP8 is an open research question. PRACTICAL IMPLICATION: if you are using FP8 activations, you may not need SmoothQuant at all - the FP8 format already handles outliers better than INT8 because the exponent provides dynamic range. But this has not been verified with a controlled benchmark. This is a priority for further research.

SmoothQuant with FP8 activations: does outlier migration still help?

Fields

Notes

Cross-references

Emulated support (2)

Native support (3)