Pretraining Large Language Models with NVFP4

FP8 training is now standard, but 4-bit floating-point TRAINING at frontier scale faces stability, convergence and implementation problems.

NVIDIA's recipe for training LLMs in the NVFP4 format: per-16-element blocks of E2M1 with an FP8 E4M3 shared scale, plus a Random Hadamard Transform integrated into the training step to control the outliers that would otherwise blow up the narrow element range. The transform is trained-in rather than post-hoc, which is the key difference from a PTQ paper.

Paper's own claim (abstract): 'a novel approach for stable and accurate training of large language models using the NVFP4 format', motivated by the 'order of tens to hundreds of yottaflops' cost of frontier pretraining. NO throughput or accuracy delta appears in the abstract, so speedup_reported is deliberately left as the stated motivation rather than a fabricated number. This is a TRAINING-efficiency paper, not an inference-speed paper.

arXiv ID 2509.25149 verified via /abs/. Venue: Semantic Scholar returns arXiv.org only - arxiv-preprint, year = v1 year 2025. Author list truncated in the arXiv page metadata; the paper is large-authored by NVIDIA. The repo already has source quant-nvfp4-paper covering the format itself and the quantization/nvfp4 record covers native Blackwell support; this paper records the TRAINING recipe, which those records do not. arXiv ID and title verified by fetching https://arxiv.org/abs/2509.25149 and reading the citation_title meta tag on 2026-10-03. Venue and year from the Semantic Scholar graph API batch endpoint (paper/arXiv:2509.25149) unless the record's notes say otherwise. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.