id
flops/paged-attention-block-table-kernel
Arithmetic intensity
Paging changes no arithmetic. It changes the memory layout so that decode attention reads a list of contiguous pages via a per-request block table instead of one dense per-request cache. FLOPs are identical and total bytes are identical; what changes is coalescing efficiency and the absence of per-request reallocation copies when the sequence grows. Its real payoff is capacity (no wasted padding in the cache) and the elimination of defragmentation copies, which is a memory-capacity and CPU-overhead win rather than a bytes-per-token win. Recorded as decode_attention.
Scales with
context_lengthbatch_sizepage size (page_block_size must be a multiple of 256 in FlashAttention's paged path, a hard kernel constraint)num_kv_heads
Hardware parameters
parametermemory_capacityeffectIts real payoff is capacity (no wasted padding in the cache) and the elimination of defragmentation copies, which is a memory-capacity and CPU-overhead win rather than a bytes-per-token win.basisunverifiedvalues_fornvidia-h100-sxm,nvidia-a100-80gb-sxm4,amd-instinct-mi300x
Workarounds
Block table indirection inside the attention kernel instead of copying blocks to make them contiguousPage size chosen as a multiple of the kernel's vectorized load width (256 in FA-2's paged path)Fused cache update: FA-2's flash_attn_with_kvcache writes the new K/V into the cache and does the attention in one kernel, avoiding a separate append kernelRotary embedding applied inside the same kernel as the cache writeBlock-sparse attention plans on top of paging to skip fully-masked pages
notes
Introduced as a kernel feature in FA-2 2.5 (credited in the README changelog). Two hard constraints visible in the source: page_block_size must be a multiple of 256, and the block_table is per (batch, max_num_blocks_per_seq) int32 - so a kernel that hard-codes page size will not accept another engine's cache. Paging is the substrate that split-KV and cascade attention both assume; it is also why a given model on the same GPU gets different decode-attention behavior between an engine with a paged cache and one with a dense cache.
HARDWARE-PARAMETER ROWS ADDED 2026-10-05 (schema authority pass, agent w5-schema). `affected_by_hardware` names PRODUCTS (accelerator record ids); these rows name the hardware QUANTITIES that move this class's cost, which had nowhere to live until flop.hardware_parameters was added in this commit. Every effect below is lifted verbatim from this record's own existing prose and no figure here is new - `magnitude` reproduces numbers the record already quoted. `basis` is 'unverified' where the effect was asserted with nothing concrete behind it, rather than being dressed up. The rows were pulled out of free-text notes; they are not a change to what this record claims.