FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

Fields

id

sources/attn-flashattention-3-paper

URL

https://arxiv.org/abs/2407.08608

Publisher

arXiv

Kind

paper

Published

2024-07-11

Accessed

2026-10-04

notes

FA-3 paper (arXiv 2407.08608). Key claims: 1.5-2.0x faster than FA-2 on H100, FP16 up to 740 TFLOPs/s (75% utilization), FP8 close to 1.2 PFLOPs/s. FA-2 achieved only 35% utilization on H100. Three techniques: warp specialization with TMA, ping-pong/2-stage interleaving, FP8 with incoherent processing (Hadamard transform). Requires H100/H800, CUDA >= 12.3.

Notes

FA-3 paper (arXiv 2407.08608). Key claims: 1.5-2.0x faster than FA-2 on H100, FP16 up to 740 TFLOPs/s (75% utilization), FP8 close to 1.2 PFLOPs/s. FA-2 achieved only 35% utilization on H100. Three techniques: warp specialization with TMA, ping-pong/2-stage interleaving, FP8 with incoherent processing (Hadamard transform). Requires H100/H800, CUDA >= 12.3.

Cross-references

This record references no other records.