id
flops/flashattention-io-aware-tiling
Arithmetic intensity
Does not raise AI arithmetically - the FLOP count of causal attention is unchanged. It removes bytes: the naive path writes the L x L score matrix to HBM and reads it back, adding Theta(L^2) HBM traffic on top of the Theta(L*d) KV traffic, which drops the effective AI from Theta(L) to well below the ridge point for most lengths. FA-1's tiling loads Q/K/V blocks HBM->SRAM once, keeps the running softmax accumulator in registers and never writes S to HBM, so bytes revert to the Theta(L*d) ideal. The claim to make precisely is 'reduces HBM traffic by removing the S x S materialization', NOT 'reduces attention FLOPs'.
Scales with
context_length (the bytes saved scale as L^2, the useful KV bytes as L)batch_size (all batch sequences become one big GEMM pair)head_dim (the S tile size per KV byte loaded)
Hardware parameters
parameterpeak_flopseffectIt removes bytes: the naive path writes the L x L score matrix to HBM and reads it back, adding Theta(L^2) HBM traffic on top of the Theta(L*d) KV traffic, which drops the effective AI from Theta(L) to well below the ridge point for most lengths.basisunverifiedvalues_fornvidia-a100-80gb-sxm4,nvidia-h100-sxm,nvidia-rtx-4090,nvidia-rtx-3090,amd-instinct-mi300xparametershared_memory_capacityeffectTiling: read K/V tiles HBM->SRAM once, accumulate softmax statistics in registers, rescale the running output when the block max changes | Recompute rather than store: the backward pass regenerates S from Q,K instead of reading it back, trading FLOPs for bytes | Shared-memory capacity forces the tile shape: a BLOCK_M x BLOCK_N score tile must fit in SMEM alongside the Q and K/V tiles, so BLOCK_N is bounded by SMEM...basisunverifiedvalues_fornvidia-a100-80gb-sxm4,nvidia-h100-sxm,nvidia-rtx-4090,nvidia-rtx-3090,amd-instinct-mi300xparametermemory_capacityeffectTiling: read K/V tiles HBM->SRAM once, accumulate softmax statistics in registers, rescale the running output when the block max changes | Recompute rather than store: the backward pass regenerates S from Q,K instead of reading it back, trading FLOPs for bytes | Shared-memory capacity forces the tile shape: a BLOCK_M x BLOCK_N score tile must fit in SMEM alongside the Q and K/V tiles, so BLOCK_N is bounded by SMEM...basisunverifiedvalues_fornvidia-a100-80gb-sxm4,nvidia-h100-sxm,nvidia-rtx-4090,nvidia-rtx-3090,amd-instinct-mi300xparametertensor_core_counteffectWhy it matters as a decision input: before FA-1, a part with weak FLOPs but strong bandwidth could not hide the S x S round trip, so long-context attention failed on memory traffic alone; after FA-1 the same part is limited by tensor-core throughput.basisunverifiedvalues_fornvidia-a100-80gb-sxm4,nvidia-h100-sxm,nvidia-rtx-4090,nvidia-rtx-3090,amd-instinct-mi300x
Workarounds
Tiling: read K/V tiles HBM->SRAM once, accumulate softmax statistics in registers, rescale the running output when the block max changesRecompute rather than store: the backward pass regenerates S from Q,K instead of reading it back, trading FLOPs for bytesShared-memory capacity forces the tile shape: a BLOCK_M x BLOCK_N score tile must fit in SMEM alongside the Q and K/V tiles, so BLOCK_N is bounded by SMEM not by theoryBlock-sparse / causal-mask skipping of fully-masked tiles (halves the useful flops on a causal mask without changing bytes)
notes
This is the mechanism every later attention kernel inherits; FA-2 through FA-4 and FlashInfer all keep the tiling and change something else on top of it. Why it matters as a decision input: before FA-1, a part with weak FLOPs but strong bandwidth could not hide the S x S round trip, so long-context attention failed on memory traffic alone; after FA-1 the same part is limited by tensor-core throughput. Speedup vs PyTorch standard attention is 2-4x wallclock per the author's recap, larger on slower GPU memory - author claim. Caveat: the AI = L flop/byte ideal assumes each KV element is read from HBM exactly once, which real kernels only approach; the deficit shows up as low tensor-core utilization at small seqlen or low occupancy.
HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.