Flash-Decoding for long-context inference

Fields

id

sources/flop-flashdecoding

URL

https://crfm.stanford.edu/2023/10/12/flashdecoding.html

Publisher

Stanford CRFM / Tri Dao

Kind

blog

Published

2023-10-12

Accessed

2026-10-03

notes

Blog describing FlashDecoding: the decode-time counterpart of FlashAttention. Splits the KV sequence across thread blocks and combines partial softmax results with a log-sum-exp merge, because a single decode step has too little parallel work to fill the GPU. Notes the resulting bandwidth-bound parallelism over the KV cache.

Notes

Blog describing FlashDecoding: the decode-time counterpart of FlashAttention. Splits the KV sequence across thread blocks and combines partial softmax results with a log-sum-exp merge, because a single decode step has too little parallel work to fill the GPU. Notes the resulting bandwidth-bound parallelism over the KV cache.

Cross-references

This record references no other records.