inference/research
draftlow · 0.50updated 2026-10-04

Hyena

recurrent
This record is a draft. Not yet verified. Provisional. Treat every number on this page as provisional.

Fields

id
models/hyena-arch
Family
hyena
Vendor
Stanford / Together AI
Release year
2023
Architecture
recurrent
Context scaling method
native
Attention variant
implicit long convolution + data-controlled gating
notes
Hyena is an architecture paper record (arXiv:2302.10866), not a released checkpoint: there is no config.json and no served weights, so no KV geometry exists to record. num_kv_heads, head_dim, gqa_ratio and kv_cache_bytes_per_token are undefined for a long-convolution operator and left null. Hyena Hierarchy per arXiv:2302.10866 (ICML 2023). Subquadratic drop-in replacement for attention: interleaves implicitly parametrized long convolutions with data-controlled gating. 'Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.' Reaches Transformer quality with 20% training compute reduction at 2K. Research-only adoption - no production engine support. [ID renamed from 'hyena-hierarchy' to 'hyena-arch' to avoid collision with papers/hyena-hierarchy. Source fixed from non-existent 'hyena' to 'pap-arch-2302-10866'.] CONTEXT_SCALING_METHOD (native): architecture record: implicit long convolution + data-controlled gating, no positional-embedding ladder at all, so there is nothing to scale.

Notes

Hyena is an architecture paper record (arXiv:2302.10866), not a released checkpoint: there is no config.json and no served weights, so no KV geometry exists to record. num_kv_heads, head_dim, gqa_ratio and kv_cache_bytes_per_token are undefined for a long-convolution operator and left null. Hyena Hierarchy per arXiv:2302.10866 (ICML 2023). Subquadratic drop-in replacement for attention: interleaves implicitly parametrized long convolutions with data-controlled gating. 'Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.' Reaches Transformer quality with 20% training compute reduction at 2K. Research-only adoption - no production engine support. [ID renamed from 'hyena-hierarchy' to 'hyena-arch' to avoid collision with papers/hyena-hierarchy. Source fixed from non-existent 'hyena' to 'pap-arch-2302-10866'.]

CONTEXT_SCALING_METHOD (native): architecture record: implicit long convolution + data-controlled gating, no positional-embedding ladder at all, so there is nothing to scale.

Cross-references

This record references no other records — it is a leaf. It is cited by 0.