inference/research
verifiedhigh · 0.85updated 2026-10-04

FP8 E4M3 (per-tensor / per-block scaled)

w8a8 (fp8 weights, fp8 activations, fp32 accumulate)

Fields

id
quantization/fp8-e4m3
Scheme
fp8-e4m3
Bits
w8a8 (fp8 weights, fp8 activations, fp32 accumulate)
Activation scheme
per-tensor scale, or per-16/32-value block scale (blockwise/rowwise variants are recorded separately as fp8-blockwise and int8-channelwise). No smoothing, no outlier decomposition.
Kernels
TensorRT-LLM FP8 (per tensor) pluginscutlass FP8 GEMMFlashAttention-3 / FA2 FP8 pathsTransformer Engine FP8FlashInfer FP8 GEMM
Quality delta
No controlled fp16 delta was located in the primary sources read for this record. Vendor checkpoint claims (Llama-3.1-8B-Instruct-FP8 et al. shipped by NVIDIA) exist but are not a perplexity comparison, so none recorded.
notes
GROUP SIZE LEFT NULL 2026-10-04 as a DEFINITIONAL point: per-tensor FP8 has one scale for the entire tensor by definition. A group size would contradict the scheme. The block-scaled variant is a separate record (fp8-blockwise) where the width IS meaningful, and the rowwise variant is a third. Any reader computing memory from this record must not assume per-32 or per-128 scaling. E4M3 (1 sign, 4 exp, 3 mantissa, max 448) is the FP8 variant chosen for *activations and weights* in the Blackwell/Hopper recipes because 3 mantissa bits preserve more relative precision; E5M2 is preferred where range matters more (gradients, and the KV cache). This is the format where the weight-vs-activation distinction flips: FP8 is genuinely w8a8 with fp32 accumulate, so the low-precision operands actually reach the tensor core. Hardware-native from Hopper on NVIDIA (H100 accelerator record lists native fp8 at 1979 TFLOPs sparse, 2x its fp16) and from CDNA3 on AMD - MI300X lists 2614.9 TFLOPs dense fp8, again 2x fp16 (quant-rocm-quant-doc confirms Quark targets this on Instinct). NOT native on Ampere (the A100 accelerator record has no fp8 entry) so FP8 on A100 is emulation by upcast. TensorRT-LLM's hardware matrix marks FP8(per tensor) Y on Hopper, Ada, Blackwell sm100/103 and Rubin, and no on Ampere (quant-trtllm-quantization).

Notes

GROUP SIZE LEFT NULL 2026-10-04 as a DEFINITIONAL point: per-tensor FP8 has one scale for the entire tensor by definition. A group size would contradict the scheme. The block-scaled variant is a separate record (fp8-blockwise) where the width IS meaningful, and the rowwise variant is a third. Any reader computing memory from this record must not assume per-32 or per-128 scaling. E4M3 (1 sign, 4 exp, 3 mantissa, max 448) is the FP8 variant chosen for *activations and weights* in the Blackwell/Hopper recipes because 3 mantissa bits preserve more relative precision; E5M2 is preferred where range matters more (gradients, and the KV cache). This is the format where the weight-vs-activation distinction flips: FP8 is genuinely w8a8 with fp32 accumulate, so the low-precision operands actually reach the tensor core. Hardware-native from Hopper on NVIDIA (H100 accelerator record lists native fp8 at 1979 TFLOPs sparse, 2x its fp16) and from CDNA3 on AMD - MI300X lists 2614.9 TFLOPs dense fp8, again 2x fp16 (quant-rocm-quant-doc confirms Quark targets this on Instinct). NOT native on Ampere (the A100 accelerator record has no fp8 entry) so FP8 on A100 is emulation by upcast. TensorRT-LLM's hardware matrix marks FP8(per tensor) Y on Hopper, Ada, Blackwell sm100/103 and Rubin, and no on Ampere (quant-trtllm-quantization).

Cross-references

Emulated support (3)

Native support (7)