FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Fields

id

sources/flop-flashattention

URL

https://arxiv.org/abs/2205.14135

Publisher

arXiv (Dao, Fu, Wolf, Rudenstein, Smith; NeurIPS 2022)

Kind

paper

Published

2022-05-26

Accessed

2026-10-03

notes

Tiling attention so Q/K/V tiles are loaded to SRAM once and reused, cutting HBM reads/writes. Claims IO complexity lower than standard attention and optimal for a range of SRAM sizes. Reports 15% end-to-end wall-clock speedup on BERT-large.

Notes

Tiling attention so Q/K/V tiles are loaded to SRAM once and reused, cutting HBM reads/writes. Claims IO complexity lower than standard attention and optimal for a range of SRAM sizes. Reports 15% end-to-end wall-clock speedup on BERT-large.

Cross-references

This record references no other records.