Skip to content
The methodOpen data
Back to the collection

Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets

Armand Joulin, Tomas Mikolov

2015 · arXiv

AlgorithmsMachine learningPsychology
Read the paper
HN score
0
Mentions
5
Readers
5
Months
4
What’s in a score?

From the abstract

Despite the recent achievements in machine learning, we are still very far from achieving real artificial intelligence. In this paper, we discuss the limitations of standard deep learning approaches and show that some of these limitations can be overcome by learning how to grow the complexity of a model in a structured way. Specifically, we study the simplest sequence prediction problems that are beyond the scope of what is learnable with standard recurrent networks, algorithmically generated sequences which can only be learned by models which have the capacity to count and to memorize sequences. We show that some basic algorithms can be learned from sequential data using a recurrent network associated with a trainable memory.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

jsenn
Neutral mention · rule
Here's a paper that shows how to augment a recurrent NN with a stack: http://arxiv.org/abs/1503.01007. I'm not an expert, so some or all of this may be wrong, but my understanding of how it works is this. You have a stack memory, which is just a list of numbers (or maybe a list of N-dimensional vectors). You add the contents of the stack at time t-1 to the input of the network at time t. The hidden layer contains a special number that controls whether to push or pop from the stack (or leave it alone). The values in the stack are determined from the values in the hidden layer at the time they were pushed and the special push/pop control parameter. The new value of the…
Read the full comment
stack is passed into the network on the next time step, and so on. The tricky thing is that the stack operations have to be differentiable, otherwise there's no way to optimize the push/pop control and the set of weights that determines how to turn the hidden layer into a stack value. So the special control variable in the hidden layer is a continuous value that controls "how much" push vs. pop to do. The value of the top of the stack is a weighted average of the values in the hidden layer and the current value on the top of the stack. So, if the push vs. pop knob is turned all the way toward pop, then the value of the stack will be whatever the second-from-the-top value is. If it's turned all the way toward push, then the value at the top will be a weighted average of the values in the hidden layer. If it's 50% push and 50% pop, then the value at the top of the stack will be 0.5 * <hidden layer value> + 0.5 * <second-from-top value>, and so on. It's hard to think about what a half push, half pop would actually mean (at least for me), but it apparently works. As a sibling commenter wrote, there are fancier architectures that provide a full differentiable RAM or other types of gadgets for the network to control.
Original comment Neural Networks and the Chomsky Hierarchy [DeepMind, 2022]
davidad_
Neutral mention · rule
Here's the differentiable stack/queue/deque paper: http://arxiv.org/abs/1506.02516 It is indeed similar to some work from FAIR, where one of the tasks was binary addition: http://arxiv.org/abs/1503.01007
Original comment Differentiable Programming
cs702
Neutral mention · rule
I have shivers going down my spine at the idea of feeding, say, skip-thought vectors representing sentences[1] into stack-augmented recurrent neural nets as proposed in Facebook's paper[2], for all sorts of interesting tasks. -- [1] http://arxiv.org/pdf/1506.06726v1.pdf [2] http://arxiv.org/pdf/1503.01007.pdf
Original comment Inferring Algorithmic Patterns with Stack

Keep following the thread.

More in Algorithms

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score