Dream 7B: Diffusion Large Language Models

Autoregressive models generate strictly sequentially, so decoding cannot be parallelised and in-context infilling or arbitrary-order generation is awkward.

A 7B diffusion language model that refines sequences in parallel through iterative denoising instead of generating left to right, achieving its results through simple training techniques rather than a new architecture: autoregressive-based LLM INITIALISATION (start from an existing AR checkpoint rather than from scratch, which is the main practical difference from LLaDA), context-adaptive token-level noise scheduling, and a diffusion backbone built on an AR LLM. Generation: begin from a masked or noised sequence, iterate the denoiser, and unmask tokens under a confidence schedule. FLOP consequence as in all masked diffusion LMs: T full-sequence forward passes for an L-token output, so FLOPs are roughly T times one full-length prefill and are much higher than autoregressive decoding; the payoff is parallel generation of many positions per step plus the flexibility of arbitrary-order generation, infilling, and a tunable quality-versus-speed tradeoff via the step count and per-step token budget. Open weights are released.

Reports consistently outperforming existing diffusion language models on general, mathematical and coding tasks, plus superior planning ability and inference flexibility including arbitrary-order generation, infilling, and tunable quality-speed trade-offs. No tok/s speedup ratio with baseline and settings is published in the abstract, so the claim being recorded is the tunable quality-speed frontier via denoising-step count, not a speedup.

arXiv preprint; no journal-ref. Engine support verified: llama.cpp carries LLM_ARCH_DREAM, lists it in llm_arch_is_diffusion(), and has a dedicated conversion/dream.py. No native Dream row appears in the vLLM supported-models list consulted. The AR-initialisation trick is the practically important detail because it means diffusion LMs no longer require the from-scratch budget LLaDA needed, which is why adoption among practitioners picked up in 2025. Model weights and code were released by the authors. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.