Large Language Diffusion Models (LLaDA)

Discrete masked diffusion LM. Section 2.4 Inference: 'The total number of sampling steps is a hyperparameter, which naturally provides LLaDA with a trade-off between efficiency and sample quality'; 'the generation length is also treated as a hyperparameter, specifying the length of the fully masked sentence at the beginning of the sampling process.' Crucially: 'We use vanilla multi-head attention instead of grouped query attention for simplicity, as LLaDA is incompatible with KV caching' and 'LLaDA does not use a causal mask, as its formulation allows it to see the entire input for predictions.' Generation length fixed at 1024 for LLaDA 8B Base in the main sampling-efficiency ablation; tuned from {64, 256, 512} for Instruct.