FlashDecoding++: Faster Large Language Model Inference on GPUs

Attention at decode time is memory-bandwidth-bound on the KV cache, and flat GEMMs of differing shapes hit different hardware bottlenecks - so a single schedule cannot be optimal across batch size, head count and context length, and doubles the cost of KV reads (the 'read amplification' of naive flash-decode splitting).

A decode-optimised attention kernel: parallelise over KV length, head and batch with a per-workload split heuristic; keep Q in registers/SRAM across the KV loop to eliminate read amplification; use different GEMV vs GEMM templates with tensor-core and adaptive pipeline scheduling; double buffering; and a heuristic dataflow that adapts to the hardware resource bottleneck given the input shape.

Paper's own claims (abstract): up to 4.86x on NVIDIA GPUs and 2.18x on AMD GPUs vs HuggingFace implementations, and an average 1.37x vs state-of-the-art LLM inference engines on mainstream LLMs. Both baselines named (HF implementations; unnamed SOTA engines - the weaker of the two comparisons).

arXiv ID 2311.01282 verified via abs-page meta tags on 2026-10-03; title matches. CORRECTION TO THE CONTEXT BRIEF AND TO THE SOURCE RECORD'S IMPLICIT FRAMING: the context brief cited 'FlashDecoding as 2210.00996' as an example of a wrong ID shipped by an earlier agent. 2210.00996 is indeed not this paper. The correct ID is 2311.01282 and the title is 'FlashDecoding++', not the original flash-decoding blog post - the dead agent's source note correctly flagged that the blog post has no arXiv paper of its own. Venue CORRECTED: pap-attn-flashdecoding says 'arXiv preprint; no journal_ref' with no venue evidence, but OpenReview/DBLP mirror records venue='MLSys 2024' for the exact title. venue=mlsys, year=2024. Adoption: the attention variants described here (flat GEMM decode splits, head/batch-length parallelism, fused decode kernel) are exactly what vLLM's flash_attn decode path and SGLang's flashattention_backend implement as the default decode kernels; the technique is upstream even though this specific repo is not the shipped code. confidence held at 0.82 rather than higher because the strongest speedup (4.86x) is against a HuggingFace reference implementation, which is a weak baseline for a serving kernel. RESOLVED (2026-10-04 cross-reference repair): 'amd-radeon-instinct-mi300x' -> amd-instinct-mi300x. The value carried the 'amd-radeon-instinct-' prefix but named the MI300X, which has exactly one record and is the AMD part behind this paper's 2.18x-on-AMD result. Not a guess about which AMD card: the datacenter MI300X and the Radeon Instinct branding are the same product family here and the repo has one MI300X record. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: mlsys has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings.