Ring Attention with Blockwise Transformers for Near-Infinite Context

Single-device attention cannot exceed a fixed sequence length on available HBM, and simply spilling sequences to many devices either breaks training parallelism or pays for communication.

Blockwise self-attention and feedforward computation are distributed across devices so that each device owns a contiguous subsequence, while the communication of key-value blocks is FULLY OVERLAPPED with the computation of blockwise attention on the ring of devices. Overlap is achieved by interleaving per-block compute and per-block communication, so the collective is hidden behind useful work.

NO SPEEDUP NUMBER IN THE ABSTRACT. The claim is about REACH rather than rate: enables training and inference of sequences up to DEVICE-COUNT TIMES longer than prior memory-efficient Transformers, without approximation and without additional communication or computation overheads, demonstrated at millions of tokens and improving performance on language modelling and RL. The zero-overhead property is the claim worth carrying.

arXiv ID 2310.01889 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. Venue verified with a title nuance: the OpenReview record's title is 'RingAttention with Blockwise Transformers for Near-Infinite Context' (no space) and returns venue='ICLR 2024 poster', venueid='ICLR.cc/2024/Conference' - same paper, same authors (Liu, Zaharia, Abbeel), spacing difference in the title only. venue=iclr, year=2024. Note two WORKSHOP versions exist at NeurIPS 2023 (Instruction and FMDM workshops, both 'Ring Attention' title) - those are precursors, the ICLR 2024 entry is the conference paper. adoption=research-only: ring/sequence-parallel attention for TRAINING is not in vLLM or SGLang, and for INFERENCE long-context decode is KV-bandwidth-bound and served by paged attention instead - a fundamentally different mechanism (recorded at pagedattention-vllm). Note the relationship to the blockwise-parallel-transformer record: same group, BPT is the single-device memory-reduction work that Ring Attention distributes, and the OpenReview record shows the BPT title under NeurIPS 2023 spotlight. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: iclr has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.