FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Fields

id

sources/attn-flashattention-2-paper

URL

https://arxiv.org/abs/2307.08691

Publisher

arXiv

Kind

paper

Published

2023-07-17

Accessed

2026-10-04

notes

FA-2 paper (arXiv 2307.08691). Key claims: ~2x speedup vs FA-1, reaching 50-73% of theoretical max FLOPs/s on A100. FA-1 reached only 25-40%. Three changes: reduced non-matmul FLOPs, parallelized across thread blocks for single head, better warp-level work distribution. Training speed up to 225 TFLOPs/s per A100 (72% MFU).

Notes

FA-2 paper (arXiv 2307.08691). Key claims: ~2x speedup vs FA-1, reaching 50-73% of theoretical max FLOPs/s on A100. FA-1 reached only 25-40%. Three changes: reduced non-matmul FLOPs, parallelized across thread blocks for single head, better warp-level work distribution. Training speed up to 225 TFLOPs/s per A100 (72% MFU).

Cross-references

This record references no other records.