DISTFLASHATTN: Distributed Memory-efficient Attention for Long-context LLMs Training

Training long-context LLMs needs the attention computation sharded across devices, and existing approaches pay for it either in sequence-length limits (attention must be exact per sequence) or in imbalance and redundant communication.

DISTFLASHATTN combines FlashAttention's tiling with sequence-level parallelism plus three named techniques: token-level workload balancing (so all devices finish together), overlapping key-value communication with compute, and a rematerialisation-aware gradient checkpointing algorithm. Same LightSeq codebase lineage as Sequence Level Parallelism.

Paper's own claims (abstract), each against a NAMED distributed baseline, on Llama-7B at sequence lengths 32K-512K: 4.45-5.64x vs Ring Self-Attention (also 8x longer sequences); 1.24-2.01x vs Megatron-LM with FlashAttention (2-8x longer sequences); 1.67x vs recent Ring Attention; and 1.26-1.88x vs DeepSpeed-Ulysses. This is the best-baselined attention record in the batch - four named competitors, a model and a length range.

arXiv ID 2310.03294 verified via abs-page citation_title/citation_date on 2026-10-03; title matches. CORRECTION TO THE SOURCE RECORD: it says 'arXiv preprint; no journal_ref' with no venue evidence, but OpenReview returns venue='COLM 2024', venueid='colmweb.org/COLM/2024/Conference' for the exact title. venue=colm, year=2024. CATEGORY AND SCOPE NOTE, deliberately emphasised: this is a TRAINING-side paper, not an inference-serving result - the source record already flags this and it matters for how the record is read. Every speedup is a training-throughput number on Llama-7B, and no figure here should be quoted as a serving latency or decode-throughput claim. adoption=research-only: no distributed-attention integration in vLLM or SGLang; sequence-parallel attention for training lives in Megatron-LM and DeepSpeed-Ulysses instead, both of which this paper benchmarks against. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: colm has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. AFFILIATIONS CORRECTED 2026-10-05 (wave 6, affiliation sweep, agent w6-papers): THREE of the five recorded institutions were not on the paper. The field previously read 'Lehigh University', 'Rice University' and 'Texas A&M University'. arXiv HTML byline for 2310.03294v2 lists five superscripted affiliations in author order - 1 UC Berkeley, 2 University of Washington, 3 UCSD, 4 CMU, 5 USC - attached to Dacheng Li, Rulin Shao, Anze Xie, Eric P. Xing, Xuezhe Ma, Ion Stoica, Joseph E. Gonzalez and Hao Zhang. NOT ONE of Lehigh, Rice or Texas A&M appears anywhere on the paper, and all five real institutions were missing. This is the SAME ERROR CLASS as the MegaScale-Infer Rice University credit that wave 4 found, in the same corpus, one wave apart, and it was missed because the two papers have nothing else in common: MegaScale-Infer was found because someone read the Crossref deposit's per-author affiliation array, while this record's affiliations were taken from a source that did not carry them and were filled from plausibility. Rice University appearing in BOTH is the tell - a plausible-looking large research university that an LLM reaches for when it has nothing to go on. Lehigh and Texas A&M are the same failure in the same record. The practical rule this establishes: an affiliation field with no byline behind it is an unverified claim about named people, and the cheapest check in the repo - read the arXiv HTML byline - is the check that catches all three. The affiliation correction does not touch the rest of the record. venue=colm/year=2024 was independently " re-confirmed in this pass via the OpenReview api2 note with venue 'COLM' and venueid " "'colmweb.org/COLM/2024/Conference' for this exact title with the Dacheng Li / Rulin Shao author overlap, and " the authors array (four names) is a truncation of the paper's eight, which is recorded here rather than left to " look like a complete list. The TRAINING-scope caveat in the note above stands: every speedup in this record is a " training-throughput number on Llama-7B at 32K-512K and none may be quoted as a serving latency or decode-" "throughput result.