Loki: Low-rank Keys for Efficient Sparse Attention (arXiv:2406.02542)

NeurIPS 2024 (per arXiv comments field: Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems Main Conference Track). Approximates self-attention by ranking and selecting tokens in the KV-cache based on attention scores computed in low-dimensional space. Key vectors lie in a significantly lower-dimensional space across datasets and models.