Ring Attention with Blockwise Transformers for Near-Infinite Context

Sequence-parallel attention: partition the KV sequence across devices and rotate blocks so that each device sees every KV block exactly once while keeping the attention math local. This is the context-length analogue of tensor parallelism and is the workload-shape precedent for treating long-context prefill as an interconnect problem once the prompt exceeds one device's memory. Useful as contrast to the all-reduce record: ring attention is P2P and needs no reduction, so its cost scales with sequence length rather than with hidden width times batch. | SOURCE AUDIT 2026-10-03: the 'publisher' field held a paragraph of prose (authors + affiliations, 55 chars) instead of a publisher name; trimmed to 'arXiv' with the authorship detail moved here.