Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention

Fields

id

sources/arch-lightning-attention

URL

https://arxiv.org/abs/2405.17381

Publisher

Qin et al. (arXiv)

Kind

paper

Published

2024-05-27

Accessed

2026-10-03

notes

Isolates the concrete reason linear attention fails to deliver its theoretical advantage in the causal setting: the cumulative summation (cumsum) required by prior linear-attention kernels. Fix is intra-block conventional attention + inter-block kernel trick plus tiling.

Notes

Isolates the concrete reason linear attention fails to deliver its theoretical advantage in the causal setting: the cumulative summation (cumsum) required by prior linear-attention kernels. Fix is intra-block conventional attention + inter-block kernel trick plus tiling.

Cross-references

This record references no other records.