Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

States the core substitution: 'we replace the majority of self-attention layers in the common Transformer model architecture with Mamba layers that perform constant computation and require constant memory per generated token'. Figure 2 caption: 'Roughly 8% of the total layers in the model are self-attention layers; these layers are evenly dispersed throughout the model.' Reports up to 3x faster at inference vs similarly sized Transformers. Also documents the FP4-on-32GiB-consumer-GPU memory budget (29.5GiB weights leaving 2.5GiB for KV cache and activations).