Hyena Hierarchy: Towards Larger Convolutional Language Models

Existing subquadratic attention replacements - low-rank and sparse approximations - have to be mixed with dense attention layers to match Transformer quality, so there is no single subquadratic operator that stands alone.

A single implicit long-convolution operator, Hyena, built by interleaving implicitly parameterised long convolutions with data-controlled gating (multiplicative gates driven by the input). The implicit parameterisation (via an MLP generating the convolution kernel weights) is what allows a single operator to have the subquadratic structure while retaining the expressivity that previously forced the hybrid design.

Paper's own claims (abstract): more than 50 points of accuracy improvement over state-space and other implicit/explicit methods on recall and reasoning at sequence lengths of thousands to hundreds of thousands of tokens; reaching Transformer quality on WikiText103 and The Pile with a 20% reduction in training compute at sequence length 2K; Hyena operators are 2x faster than highly optimised attention at sequence length 8K and 100x faster at sequence length 64K. The speedup figures are operator-level with explicit sequence lengths, which is the right kind of claim.

arXiv ID 2302.10866 verified via abs-page meta tags on 2026-10-03; title matches. Venue CORRECTED from the source record's 'arXiv preprint; no journal_ref': OpenReview returns venue='ICML 2023 OralPoster', venueid='ICML.cc/2023/Conference' for the exact title. venue=icml, year=2023. adoption=research-only verified by absence: Hyena is in transformers' HYBRID_ATTENTION_MODELS_TO_HYBRID_CACHE_MAPPING-style model lists at most indirectly - no Hyena model file in vLLM's registry, none in SGLang's 248-file models directory, and no Hyena cache layer in transformers cache_utils.py (which has the linear-attention layers that serve the Mamba/RWKV/BASED family but no Hyena). The 100x-at-64K operator claim is real and worth keeping, but an operator speedup at 64K says nothing about decode throughput, where Hyena's recurrence would have to compete with SSM kernels that ARE shipped. VENUE_TRACK BACKFILL 2026-10-04 (paper-fields pass, agent w3p): venue_track 'main'. This is STRUCTURAL rather than researched: icml has no Findings volume and no workshop track in which a paper could appear, so 'main' is the only possible honest value and no per-paper evidence was needed. The field is annotated here so that 'findings' and 'workshop' remain MEANINGFUL - a filter for findings=true must return only the papers that actually are findings, which requires every other paper to be explicitly NOT findings. DUPLICATE-SLUG-CONSOLIDATION-2026-10-05 (agent w6-papers). data/papers/hyena.json and data/papers/hyena-hierarchy.json were BOTH live records for arXiv 2302.10866 (Hyena Hierarchy: Towards Larger Convolutional Language Models), so one paper appeared twice under two slugs and - where the records disagreed - under two different venues. hyena-hierarchy.json has been merged into this record and removed. IDENTIFICATION IS BY ARXIV ID, RE-FETCHED NOT ASSUMED: the arXiv API returns one entry for 2302.10866 with this title and the author list [Poli, Michael, Massaroli, Stefano, Nguyen, Eric, Fu, Daniel Y.], which both records were matching. Source w6p-arxiv-api-duplicate-id-confirmation. Venue evidence per record is in w6p-openreview-api2-duplicate-venue-records. Every arXiv id in this consolidation was taken from an API response rather than from memory or from the records, per the rule that this repo has shipped wrong ids twice. WHAT WAS CHECKED BEFORE REMOVING ANYTHING: every field of every data/**/*.json record was scanned for the string "hyena-hierarchy". Zero inbound references were found - no model, gotcha, source, engine or doc record pointed at it - so the merge breaks no reference. Papers are cited by SOURCE ID, never by slug, so no `sources` entry needed repointing; the retired record's source ids were folded into this record's `sources` so the evidence survives the merge. MERGED FIELD BY FIELD - nothing was silently dropped: mechanism: the retired record's text was materially fuller (952 chars vs 416) and is RETAINED here rather than discarded. Its text: A subquadratic drop-in replacement for the attention operator built by interleaving implicitly parameterised long convolutions with data-controlled gating. The operator alternates projection, short convolution and a long implicit filter; the 'hierarchy' means applying this Hyena operator repeatedly, with increasing sequence lengths, in a pyramid so that early layers see short windows and later layers see the full sequence. The long convolution is computed in frequency space with an FFT rather than materialised as a matrix, giving O(L log L) instead of O(L^2), and the filter is implicitly parameterised (as an MLP over frequency) rather than stored per-length. The resulting state is a fixed-size convolution state, CONSTANT in sequence length, so there is no KV cache. What makes it relevant to this slice is that it is one of the clean demonstrations that a convolution-plus-gating operator can reach long-range recall without attention at all. hardware_relevance UNION taken: this record had A100 40GB SXM4 and A100 80GB SXM4, the retired twin had A100 80GB SXM4 and H100 SXM. Disjoint additions, no contradiction. open_weights: open_weights was null here and true on the retired twin; github.com/HazyResearch/hyena is a public repository cited by this record. AFFILIATIONS CORRECTED - AFFILIATION ERROR FOUND AND CORRECTED. arXiv HTML byline for 2302.10866v3 carries exactly two affiliation markers: '^1Stanford University' and '^2Mila and Universite de Montreal'. This record listed Stanford, Princeton AND Carnegie Mellon; Princeton and CMU appear nowhere on the paper - Bengio's Mila affiliation is the second marker. The retired twin listed 'Stanford University' and 'NVIDIA', which is also wrong: NVIDIA funded the work but no NVIDIA-affiliated author is on the byline. So both records were wrong and they disagreed with each other, which is the clearest possible demonstration of why duplicate records are worse than no record. VENUE AND THE CLAIMS THAT COMPETED FOR IT: ICML 2023 on both records. The retired twin cited the arXiv COMMENT FIELD for it; this record cites the OpenReview note with venue 'ICML 2023 OralPoster' and venueid 'ICML.cc/2023/Conference' for the exact title, corroborated on the Poli/Massaroli/Nguyen/Fu author list. The comment-field-only evidence is demoted and the explicit venueid promoted - the same standard that made the S4 JMLR claim unacceptable. STATUS: the contradiction this pair represented is now resolved, so the record is no longer marked contested on account of the duplicate. Fields were changed as stated above; arxiv_id is unchanged.