inference/research
verifiedmedium · 0.65updated 2026-10-04

INT8 W8A8 rowwise / per-channel

w8a8

Fields

id
quantization/int8-channelwise
Scheme
int8-channelwise
Bits
w8a8
Activation scheme
per-row (per-output-channel) activation scales computed on the fly; weights scaled per output channel. Rowwise scaling rather than per-tensor is what keeps outliers from dominating.
Kernels
TensorRT-LLM FP8(rowwise)/INT8 rowwise GEMM pluginscutlass rowwise-scaled GEMMAMD Quark
notes
GROUP SIZE LEFT NULL 2026-10-04: this record is per-ROW / per-output-channel. One scale per whole output channel row, which is the maximum possible group along the reduction axis - there is no smaller grouping to report. Recording 32 or 128 would misdescribe it. Rowwise (per-output-channel) scaling is the modern default for 8-bit serving and is a separate recipe from per-tensor: the TensorRT-LLM support matrix lists 'FP8(rowwise)' and 'FP8(per tensor)' as distinct columns and they have different hardware coverage - rowwise is Y on Ada, Hopper, Blackwell and Rubin, while block scaling is *not* Y on Ada (quant-trtllm-quantization). For INT8 the analogue is the rowwise path. Recorded separately from int8 because the granularity, not just the bit-width, is what determines whether a kernel exists. No fp16 delta recorded - AMD's Quark documentation describes the capability and available FP8 checkpoints but publishes no controlled perplexity comparison.

Notes

GROUP SIZE LEFT NULL 2026-10-04: this record is per-ROW / per-output-channel. One scale per whole output channel row, which is the maximum possible group along the reduction axis - there is no smaller grouping to report. Recording 32 or 128 would misdescribe it. Rowwise (per-output-channel) scaling is the modern default for 8-bit serving and is a separate recipe from per-tensor: the TensorRT-LLM support matrix lists 'FP8(rowwise)' and 'FP8(per tensor)' as distinct columns and they have different hardware coverage - rowwise is Y on Ada, Hopper, Blackwell and Rubin, while block scaling is *not* Y on Ada (quant-trtllm-quantization). For INT8 the analogue is the rowwise path. Recorded separately from int8 because the granularity, not just the bit-width, is what determines whether a kernel exists. No fp16 delta recorded - AMD's Quark documentation describes the capability and available FP8 checkpoints but publishes no controlled perplexity comparison.

Cross-references