HGRN2: Gated Linear RNNs with State Expansion

Fields

id

sources/arch-hgrn2

URL

https://arxiv.org/abs/2404.07904

Publisher

Qin et al. (arXiv)

Kind

paper

Published

2024-04-11

Accessed

2026-10-03

notes

HGRN2 enlarges the recurrent state via an outer product with NO additional parameters and gains a linear-attention interpretation, letting it reuse GLA's chunkwise kernels. Explicitly contrasts fine-grained per-channel decays (HGRN2/GLA) with shared-per-head decays (Mamba2/xLSTM/Gated Retention): shared decays are more tensor-core-friendly but less fine-grained.

Notes

HGRN2 enlarges the recurrent state via an outer product with NO additional parameters and gains a linear-attention interpretation, letting it reuse GLA's chunkwise kernels. Explicitly contrasts fine-grained per-channel decays (HGRN2/GLA) with shared-per-head decays (Mamba2/xLSTM/Gated Retention): shared decays are more tensor-core-friendly but less fine-grained.

Cross-references

This record references no other records.