Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (arXiv:2404.02258)

arXiv:2404.02258. Mixture-of-Depths: Dynamically allocating compute in transformer-based language models. By David Raposo et al. (Google DeepMind). Submitted 2024-04-02. Dynamically allocates compute (FLOPs) to different layers based on token importance. 2026-10-10: Fetched https://arxiv.org/abs/2404.02258. Title: 'Mixture-of-Depths: Dynamically allocating compute in transformer-based language models'. Preprint matching record claims; abstract confirms learning dynamic allocation of FLOPs to specific token positions in sequence.