NVFP4 Trains with Precision of 16-Bit and Speed and Efficiency of 4-Bit

NVIDIA blog on NVFP4 pretraining. Key facts: NVFP4 pretraining recipe combines micro-block scaling, E4M3 scale factors, Hadamard transforms, selective 2D block-based quantization, and stochastic rounding. 12B Hybrid Mamba-Transformer model trained on 10 trillion tokens shows validation loss closely matching FP8 baselines. Blackwell Ultra delivers 7x GEMM speedup over Hopper for FP4 precision. GB200 and GB300 enable efficient 4-bit training.