FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Fields

id

sources/attn-flashattention-paper

URL

https://arxiv.org/abs/2205.14135

Publisher

arXiv

Kind

paper

Published

2022-05-27

Accessed

2026-10-04

notes

FA-1 paper (arXiv 2205.14135). Key claims: 2-4x wallclock speedup vs PyTorch standard attention. 15% end-to-end on BERT-large, 3x on GPT-2, 2.4x on long-range arena. IO-aware tiling eliminates S x S HBM round trip. Block-sparse extension for approximate attention. First to achieve better-than-chance on Path-X (16K) and Path-256 (64K).

Notes

FA-1 paper (arXiv 2205.14135). Key claims: 2-4x wallclock speedup vs PyTorch standard attention. 15% end-to-end on BERT-large, 3x on GPT-2, 2.4x on long-range arena. IO-aware tiling eliminates S x S HBM round trip. Block-sparse extension for approximate attention. First to achieve better-than-chance on Path-X (16K) and Path-256 (64K).

Cross-references

This record references no other records.