Retentive Network: A Successor to Transformer for Large Language Models

Retention has three equivalent computation paradigms: parallel (training), recurrent (O(1) inference), chunkwise recurrent. Measured on A100-80GB with a 6.7B model: GPU memory down 3.4x, throughput up 15.6x, latency down 8.4x at 8k input; 'For a 7B model and 8k sequence length, RetNet decodes 8.4x faster and saves 70% of memory than Transformers with key-value caches.' Also reports 'inference latency is insensitive to batch size'.