Blockwise Parallel Transformer for Large Context Models

Self-attention memory and the feedforward layer's activation memory both limit the sequence length a single device can train on, so long-sequence tasks need awkward chunking or approximations.

Blockwise computation of self-attention combined with FUSION of the feedforward network into the attention block, so the activation memory for both is bounded by one block rather than by the full sequence. Attention within a block is computed exactly; the block decomposition changes memory, not the attention arithmetic. This is the single-device work whose device-level distribution became Ring Attention.

Paper's own claim (abstract): enables training sequences 32x LONGER than vanilla Transformers and up to 4x longer than previous memory-efficient methods. This is a CAPACITY claim (max trainable sequence length), not a throughput or latency speedup - the mechanism reduces memory, and the number expresses how much sequence that buys. Quoting '32x' as a speedup would misstate it.

arXiv ID 2305.19370 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. VENUE CORRECTED from the source record's 'arXiv preprint; no journal_ref': the OpenReview record with this exact title and authors (Liu, Abbeel) returns venue='NeurIPS 2023 spotlight', venueid='NeurIPS.cc/2023/Conference'. Note a title nuance - OpenReview lists it as 'Blockwise Parallel Transformers for Large Context Models' (plural Transformers) and the arXiv title is singular; same paper. venue=neurips, year=2023. adoption=research-only: BPT is a training-memory technique; the single-device idea it embodies is now handled inside vLLM's paged KV block abstraction (recorded at pagedattention-vllm), but BPT as a training method is not in vLLM or SGLang. Pair with ring-attention (same group, distributed version) when reading the blockwise family. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: neurips has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.