Pretraining Large Language Models with NVFP4 (arXiv 2509.25149)

NVIDIA-authored paper on NVFP4 pretraining. Key facts: NVFP4 recipe integrates Random Hadamard transforms (RHT) to bound block-level outliers, two-dimensional quantization scheme for forward/backward consistency, stochastic rounding for unbiased gradient estimation, selective high-precision layers. 12B model trained on 10 trillion tokens - longest publicly documented 4-bit training run. Training loss and downstream task accuracies comparable to FP8 baseline. Authors: NVIDIA + 89 co-authors.