Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

An LLM serving long or unbounded input either pays quadratic cost or must truncate, and the usual compressions (sliding window, sparse attention) lose the distant context that mattered.

Infini-attention: build a compressive memory into vanilla attention, so ONE Transformer block carries masked LOCAL attention plus long-term LINEAR attention against a bounded compressed memory (a continuous-weight causal summary rather than a discrete token summary). Memory and computation stay bounded by the compression size regardless of input length, which is what makes streaming inference possible.

NOT A THROUGHPUT CLAIM. The abstract reports effectiveness on long-context language modelling, 1M-length passkey block retrieval and 500K-length book summarisation at 1B and 8B, plus 'minimal bounded memory parameters' and 'fast streaming inference'. No multiplier, no baseline engine. The bounded-memory property is the architectural claim worth carrying.

arXiv ID 2404.07143 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. venue=arxiv-preprint: arXiv comment is only '9 pages, 4 figures, 4 tables' and OpenReview returns DBLP 'CoRR 2024' with no conference venueid. adoption=research-only verified by absence in the same way as the others: no infini-attention model, kernel or cache layer in vLLM's registry or v1 backends, SGLang's models directory, or transformers cache_utils.py. Note the arXiv v2 note in the source record (added background and acknowledgments) - relevant only in that it means the v1 numbers and the v2 text are not identical documents. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'unknown'. `venue` is 'arxiv-preprint', which means no peer-reviewed venue has been established for this work as of the verification date, so the track genuinely is not known. 'unknown' is used here rather than leaving the field null or guessing 'main': the schema separates 'unknown' from 'main' precisely so that an unprereviewed preprint is not silently counted as main-track work. This annotation is what lets a downstream filter distinguish 'peer reviewed, main track' from 'not peer reviewed' without inferring it from the venue string. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/infini-attention.json and data/papers/infini-attention-infinite-context.json were BOTH live records for arXiv 2404.07143 (Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. infini-attention-infinite-context.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2404.07143 with this title and the author list [Munkhdalai, Tsendsuren, Faruqui, Manaal, Gopal, Siddharth], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "infini-attention-infinite-context". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the two records carried DIFFERENT text for the same paper. This record's text is retained and the retired record's variant is quoted so the difference is visible: "Infini-attention incorporates a COMPRESSIVE MEMORY into vanilla attention and builds BOTH masked local attention AND long-term LINEAR attention mechanisms inside a SINGLE transformer block. The compressive memory keeps a bounded-size persistent state, so memory and compute per step are constant in sequence length rather than linear, while the local window preserves exact short-range behaviour. Enables fast STREAMING inference because no new parameters or state resizing are needed as the stream grows." speedup_reported: the retired record's text was materially fuller (488 chars vs 356) and is RETAINED here rather than discarded. Its text: Claims minimal bounded memory parameters and fast streaming inference, demonstrated on long-context language-modeling benchmarks, 1M-SEQUENCE-LENGTH passkey context-block retrieval, and 500K-LENGTH book summarisation, with 1B and 8B models. NO numeric throughput or latency multiplier is quoted in the abstract - the claim is bounded memory plus feasibility at 1M/500K lengths, so speedup_reported records the qualitative claim and the evaluation scale rather than inventing a multiplier. hardware_relevance deliberately left EMPTY rather than taking the retired twin's 'nvidia-a100-sxm'. That id does not exist in data/accelerators/ - the repo has nvidia-a100-40gb-sxm4 and nvidia-a100-80gb-sxm4 - so the twin's entry was a DANGLING REFERENCE, one of the 31 in the committed debt list. The paper names no GPU, so the empty list is both accurate and one reference cleaner; retiring the twin removes that dangling ref outright and lowers the count validate.py reports. AFFILIATIONS CORRECTED - arXiv HTML byline for 2404.07143v2 carries one affiliation string, 'Google', with a google.com contact address. This record listed 'Google Research' AND 'Google DeepMind'; DeepMind is not named on this paper. The retired twin's plain 'Google' matched the byline exactly. VENUE AND THE CLAIMS THAT COMPETED FOR IT: No peer-reviewed venue, agreed by both records. The arXiv comment is only a page/figure/table count ('9 pages, 4 figures, 4 tables') and the OpenReview mirror returns DBLP 'CoRR 2024' with no conference venueid. Per the repo rule that comment-field-only evidence yields arxiv-preprint - and here there is not even a comment. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.