FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Fields

id

sources/flop-flashattention-2

URL

https://arxiv.org/abs/2307.08691

Publisher

arXiv (Tri Dao)

Kind

paper

Published

2023-07-17

Accessed

2026-10-03

notes

States FlashAttention-1 reached only 25-40% of theoretical maximum FLOP/s due to poor work partitioning between thread blocks and warps (low occupancy or redundant shared-memory traffic). FA-2 fixes partitioning and reduces shared-memory reads/writes.

Notes

States FlashAttention-1 reached only 25-40% of theoretical maximum FLOP/s due to poor work partitioning between thread blocks and warps (low occupancy or redundant shared-memory traffic). FA-2 fixes partitioning and reduces shared-memory reads/writes.

Cross-references

This record references no other records.