Skip to content
The methodOpen data
Back to the collection

Resurrecting Recurrent Neural Networks for Long Sequences

Antonio Orvieto, Samuel L Smith, Albert Gu

Show all 7 authors

Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, Soham De

2023 · arXiv

Machine learning
Read the paper
HN score
1
Mentions
5
Readers
4
Months
4
What’s in a score?

From the abstract

Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have the added benefits of fast parallelizable training and RNN-like fast inference. However, while SSMs are superficially similar to RNNs, there are important differences that make it unclear where their performance boost over RNNs comes from. In this paper, we show that careful design of deep RNNs using standard signal propagation arguments can recover the impressive performance of deep SSMs on long-range reasoning tasks, while also matching their training speed. To achieve this, we analyze and ablate a series of changes to standard RNNs including linearizing and diagonalizing the recurrence, using better parameterizations and initializations, and ensuring proper normalization of the forward pass. Our results provide new insights on the origins of the impressive performance of deep SSMs, while also introducing an RNN block called the Linear Recurrent Unit that matches both their performance on the Long Range Arena benchmark and their computational efficiency.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

tsurba
Recommendation · rule
Proper initialization of layers keeps gradient magnitudes from vanishing/exploding in deep networks. If you make sure the output of each layer has mean 0, std 1, the gradients will be reasonable as well, for example. I recommend e.g. the og resnet paper and its follow-up from Kaiming He et al. For a modern take on RNNs, read https://arxiv.org/abs/2303.06349 by DeepMind. There essentially the point is that largest eigenvalue (spectral radius) needs to be around 1, meaning repeated applications of a linear transformation doesn’t cause increase or decrease of the activations.
Original comment Transformers Without Normalization
scarmig
Neutral mention · rule
Nonlinearity somewhere is fundamental, but it doesn't need to be between each layer. You can, for instance, project each input to a higher dimensional space with a nonlinearity, and the problem becomes linearly separable with high probability (cf Cover's Theorem). So, for XOR, (x, y) -> (x, y, xy), and it becomes trivial for a linear NN to solve. Architectures like Mamba have a linear recurrent state space system as their core, so even though you need a nonlinearity somewhere, it doesn't need to be pervasive. And linear recurrent networks are surprisingly powerful (https://arxiv.org/abs/2303.06349, https://arxiv.org/abs/1802.03308).
Original comment Were RNNs all we needed?
smaddox
Neutral mention · rule
> Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep sequences. What would you call "ultra-deep"? [1] shows how to train an RNN in a GPT-like mode, using parallel scan, with great performance on Path-X, which has a sequence length of 16k. It's based on prior papers doing the same thing but from a state space model perspective. [1] https://arxiv.org/abs/2303.06349
Original comment Scaling Transformers to 1B Tokens
euclaise
Neutral mention · rule
> The bitter lesson [1] is going to eventually come for all of these. Eventually we'll figure out how to machine-learn the heuristic rather than hard code it. Recurrent neural networks (RNNs) do this implicitly, but we don't yet know how to effectively train RNNs on ultra-deep sequences. Linear RNNs and RWKV are examples of RNNs on deep sequences: https://arxiv.org/abs/2303.06349 https://arxiv.org/abs/2305.13048
Original comment Scaling Transformers to 1B Tokens
smaddox
Neutral mention · rule
Other approaches for long sequence lengths: - "Hyena Hierarchy: Towards Larger Convolutional Language Models" https://arxiv.org/pdf/2302.10866.pdf - "Resurrecting Recurrent Neural Networks for Long Sequences" https://arxiv.org/pdf/2303.06349.pdf It's unclear (at least to me) if any of these scale as well as transformers at this point.
Original comment From Deep to Long Learning

Keep following the thread.

More in Machine learning

More from the same topics.