TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer

Linear-attention LLMs were claimed to beat softmax attention on accuracy, but in practice suffered attention dilution, unstable training, and no practical inference path.

Evolves TransNormer with five changes: (1) LRPE - a relative position embedding - combined with exponential decay to prevent attention dilution while retaining global token interaction; (2) Lightning Attention to accelerate the linear attention kernel (>2x runtime, 4x less memory); (3) a gating mechanism for training stability; (4) a new tensor-normalisation scheme; (5) an inference algorithm that keeps decode speed constant regardless of sequence length, plus a model-parallel schema for large clusters.

Paper's own claims (abstract), component-wise with the comparison named in each case: Lightning Attention accelerates linear attention by more than 2x runtime and reduces memory 4x (baseline: the linear-attention formulation it replaces); the tensor-normalisation scheme yields over 20% acceleration; trained model sizes 385M, 1B and 7B on a self-collected corpus, matching SOTA Transformer-based LLMs while being faster.

arXiv ID 2307.14995 verified via abs-page meta tags on 2026-10-03; title matches. VENUE - and this is a correction to a widely repeated citation: TransNormerLLM is NOT ICLR 2024. OpenReview forum OROKjdAfjs (authors Qin, Li, Sun, Sun - matching the paper) returns venueid='ICLR.cc/2024/Conference/Rejected_Submission', venue='Submitted to ICLR 2024'. venue=arxiv-preprint on evidence, with this note. adoption=abandoned, and the basis is a live check: official repo OpenNLPLab/TransnormerLLM last pushed 2024-01-23 with 255 stars; no TransNormer entry in vLLM's model registry, SGLang's 248-file models directory, or transformers models/; and the sibling lightning-attention repo is equally stalled. The paper's own 'seamless deployment on large-scale clusters enabling 175B-parameter expansion' is a design intent in the paper, not a shipped deployment - worth stating because the abstract reads as if it were done. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string.