Skip to content
The methodOpen data
Back to the collection

Hyena Hierarchy: Towards Larger Convolutional Language Models

Michael Poli, Stefano Massaroli, Eric Nguyen

Show all 9 authors

Daniel Y. Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, Christopher Ré

2023 · arXiv

Language modelsComputer visionMachine learning
Read the paper
HN score
0
Mentions
7
Readers
7
Months
4
What’s in a score?

From the abstract

Recent advances in deep learning have relied heavily on the use of large Transformers due to their ability to learn at scale. However, the core building block of Transformers, the attention operator, exhibits quadratic cost in sequence length, limiting the amount of context accessible. Existing subquadratic methods based on low-rank and sparse approximations need to be combined with dense attention layers to match Transformers, indicating a gap in capability. In this work, we propose Hyena, a subquadratic drop-in replacement for attention constructed by interleaving implicitly parametrized long convolutions and data-controlled gating. In recall and reasoning tasks on sequences of thousands to hundreds of thousands of tokens, Hyena improves accuracy by more than 50 points over operators relying on state-spaces and other implicit and explicit methods, matching attention-based models. We set a new state-of-the-art for dense-attention-free architectures on language modeling in standard datasets (WikiText103 and The Pile), reaching Transformer quality with a 20% reduction in training compute required at sequence length 2K. Hyena operators are twice as fast as highly optimized attention at sequence length 8K, and 100x faster at sequence length 64K.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

pico_creator
Neutral mention · rule
It is important to note the claims are made for similar dataset & params. If you like some independent verification outside eleuther circle. There is microsoft paper on retnet: https://arxiv.org/abs/2307.08621 And the stanford heyena paper: https://arxiv.org/abs/2302.10866 Do note that both cited are competing alternatives for linear transformers, and in some sense competes with RWKV, in this upcoming "linear transfromer architecture" trend. --- While all the above validates architecture. It's important to separate out architecture design from dataset. The quality of an AI model answers, is equal parts as much about the dataset, than the architecture used. This is why it makes less sense comparing LLaMA2 directly with retnet/RWKV for example, due to the large differences in dataset size. > Disclaimer: I am the same Eugene in the podcast
Original comment The Eleuther AI Mafia
hervature
Neutral mention · rule
I never know if I have an inside scoop or an outside scoop. Has Hyena not addressed the scaling of context length [1]? I know this version is barely a month old but it was shared to me by a non-engineer the week it came out. Still, giving interviews where the person takes away that the main limitation is context length and requires a big breakthrough that already happened makes me seriously question whether or not he is qualified to speak on behalf of OpenAI. Maybe he and OpenAI are far beyond this paper and know it does not work but surely it should be addressed? [1] - https://arxiv.org/pdf/2302.10866.pdf
Original comment OpenAI's plans according to sama
saurabh20n
Neutral mention · rule
Notes from quick read of paper at https://arxiv.org/abs/2302.10866. Title of popsci is overreaching, this is a drop-in subquadratic replacement for attention. Could be promising, but to be seen if it is adopted in practice. skybrian (https://news.ycombinator.com/item?id=35657983) points out new blog post by authors, and prev discussion of older (march 28th) blog post. Takeaways: * In transformers, cost scales quadratically with length of sequence, which restricts model context. This work presents subquadratic exact operator allowing it to scale to larger contexts (100k+). * They introduce an operator called "Hyena hierarchy", a recurrence over 2 subquadratic operations:…
Read the full comment
long convolution, and element-wise mul gating. Sec 3.1-3.3 define the recurrences, matrices, and filters. This is importantly, a drop in replacement for attention. * Longer context: 100x speedup over FlashAttention at 64k context (if we view flash attention as an non-approx engg optimization, then this work is improving algorithmically, and getting OOM over that). Associate recall, i.e., just pull data, show improvements: Experiments on 137k context, and vocab sizes of 10-40 (unsure why they have bad recall on small length sequence with larger vocab, but they still outperform others) * Comparisons (on relatively small models, but hoping to show pattern) with RWKV (attention-free model, trained on 332B tokens), GPTNeo (trained on 300B tokens), with Hyena trained on 137B tokens. Models are 125M-355M sized. (Section 4.3) * On SuperGLUE, zero-shot and 3-shot accuracy is ballpark similar to GPTNeo (although technically they underperform a bit for zero-shot and overperform a bit for 3-shot). (Table 4.5 and 4.6) * Because they can support large (e.g., 100k+) context, they can do image classification. They report ballpark comparable against others. (Table 4.7) Might have misread some takeaways; happy to be corrected.
Original comment New technology could blow away GPT-4 and everything like it
smaddox
Neutral mention · rule
Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf - "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf It's unclear (at least to me) if any of these scale as well as transformers at this point.
Original comment From Deep to Long Learning
jmole
Neutral mention · rule
Oddly enough, I was reading their paper just last night: https://arxiv.org/pdf/2302.10866.pdf I think we're going to see a lot more in the wavelet/convolution/fft space when thinking about how to increase context length. I think there's also a lot of room for innovation in the positional encoding and how it's represented in transformer models, it seems like people have been trying lots of things and going with what works, but most of it is like: "look, a new orthonormal basis!". Hyena sort of seems like the first step in moving to positional embeddings (or joint positional/attentional embeddings). Very cool work.
Original comment From Deep to Long Learning

Keep following the thread.

More in Language models

More from the same topics.