Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention

Three attention paths per query: compressed coarse-grained tokens, top-k selected fine-grained tokens, and a sliding window. Key serving-cost statements: (a) 'attention computation with softmax architectures accounts for 70-80% of total latency when decoding 64k-length contexts'; (b) token-granular selection 'leads to the need to load a large number of individual tokens from the KV cache during attention computation. This non-contiguous memory access prevents efficient adaptation of fast attention techniques'; (c) their kernel achieves 'near-optimal arithmetic intensity by (1) eliminating redundant KV transfers through group-wise sharing, and (2) balancing compute workloads across GPU streaming multiprocessors'; (d) at 64k length, speedups of 11.6x / 9.0x / 6.0x on decode / forward / backward.