Differential Transformer

Softmax attention is a convex combination of value vectors, so noise or distracting context in the values is averaged into every output - the mechanism by which irrelevant context degrades key-information retrieval and induces hallucination.

Differential attention: compute attention as the difference of two softmax attention maps - lambda times softmax(Q1 K1^T) V1 MINUS softmax(Q2 K2^T) V2 - so the subtraction cancels the noise-like component and leaves the signal. Implemented with efficient kernel-level support so the doubled attention cost is manageable; the subtraction also reduces activation outliers, which matters for quantisation.

NOT A PERFORMANCE CLAIM. The abstract reports quality and robustness benefits: long-context modelling, key information retrieval, hallucination mitigation in QA and summarisation, more accurate AND more permutation-robust in-context learning, and reduced activation outliers. No speedup figure, no baseline engine. The architecture does more work than softmax attention (two attention maps instead of one), so the honest framing is a quality/robustness trade, not a free win.

arXiv ID 2410.05258 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. Venue confirmed twice: arXiv comment 'Accepted as an Oral Presentation at ICLR 2025' and OpenReview forum OvoCm1gGhN returning venue='ICLR 2025 Oral', venueid='ICLR.cc/2025/Conference' for the exact title. venue=iclr, year=2025. adoption=research-only verified by absence: no diff-transformer or differential-attention model in vLLM's registry, none in SGLang's 248-file models directory, none in transformers models/. Note one adjacent item so it is not mistaken for adoption: vLLM ships vllm/v1/attention/backends/flash_attn_diffkv.py, but diffkv (differential KV cache, as in Dream/LLaDA-style models) is a different mechanism from this paper's differential ATTENTION. open_weights=false: no released checkpoint of a differential-attention model. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.