Large Language Diffusion Models (LLaDA)

LLM capability was treated as depending on autoregressive modelling, so the entire premise was untested at 8B scale.

Trains an 8B-parameter diffusion language model FROM SCRATCH, not an adaptation of an AR model, under the standard pretraining plus SFT paradigm. Forward masking process plus reverse generation process, both parameterised by a plain Transformer that predicts masked tokens, optimised by maximising a variational lower bound on the likelihood. Inference is iterative: starting from an all-mask sequence, each step the model predicts a distribution over every masked position, a subset is unmasked according to the confidence-based remasking schedule, and this repeats until the sequence is complete. The FLOP CONSEQUENCE is the defining one for this whole record type: decoding is NOT token-by-token. Producing an L-token answer costs T full-sequence forward passes (T is typically set in the hundreds for quality), each costing about the same as one prefill of length L, so total decode compute is roughly T times a single full-length prefill. That is far more FLOPs than autoregressive decoding of L tokens, and the win is not FLOPs but wall-clock at high batch parallelism: many sequences decode simultaneously with no sequential dependency, and tokens are emitted in blocks. There is no KV cache in the usual sense, since each denoising step is recomputed from scratch over the full sequence rather than extended.

No throughput speedup. The paper's claims are quality and scaling: strong scalability across general tasks, mathematics, code and more, performing comparably to the authors' self-constructed autoregressive baselines at the same size, plus instruction-following and question-answering results. The release is also framed around a token-generation speed figure on their own hardware, but that is a vendor measurement of an unreleased-at-scale system and not reproduced here.

arXiv preprint (v3 2025-10-18); no journal-ref. Title has no colon; the 8B model is LLaDA and the arXiv ID is 2502.09992 (note the 2502, not 2501: several aggregators misdate this). Engine support is genuinely real and was the surprise in this slice: llama.cpp carries LLM_ARCH_LLADA, LLM_ARCH_LLADA_MOE and lists them in llm_arch_is_diffusion(), with a dedicated conversion/llada.py, so a full diffusion LM is first-class in a mainstream engine. vLLM has NO native LLaDA row in the supported-models list consulted, though an open PR requests one, so on vLLM it runs only via the Transformers modeling backend or trust_remote_code. Diffusion LMs are therefore NOT research-only any more, which contradicts the common assumption. VENUE CORRECTION + VENUE_TRACK 2026-10-04 (paper-fields pass, agent w3p): venue 'arxiv-preprint' -> 'neurips', venue_track 'main'. EVIDENCE: the STATIC proceedings index for this venue lists this exact title. Venue 'NEURIPS' paper page: NeurIPS 2025 poster 118608. The listing was corroborated by author overlap, not by title alone, because these titles are duplicated across venues and years and a title-only match is exactly the kind of thing that produced this repo's two previously-shipped wrong arXiv IDs. BOTH VALUES RETAINED PER THE REPO CONTRACT: the prior value was 'arxiv-preprint'. status set to contested because this changes a peer-reviewed attribution. arxiv_id unchanged (2502.09992). Method note: OpenReview's api2 hard-429s under load and pre-2024 venues are unreachable through it, so these were checked against static proceedings indexes (iclr.cc/virtual/<year>/papers.html and neurips.cc/virtual/<year>/papers.html, with the per-paper JSON-LD creditText field as the venue-year assertion). 2026-10-10: NeurIPS 2025 accepted titles index (idx7.py: https://nips.cc/virtual/2025/papers.html?filter=titles) and ICLR 2025 check (idx3.py) examined "Large Language Diffusion Models". arXiv API 2502.09992 (axc.py) comment is null. Source pap-arch-2502-09992 cites arXiv preprint vs NeurIPS 2025. Sources disagree; status remains contested.