Skip to content
The methodOpen data
Back to the collection

One-shot Learning with Memory-Augmented Neural Networks

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, Timothy Lillicrap

2016 · arXiv

Machine learning
Read the paper
HN score
0
Mentions
7
Readers
7
Months
4
What’s in a score?

From the abstract

Despite recent breakthroughs in the applications of deep neural networks, one setting that presents a persistent challenge is that of "one-shot learning." Traditional gradient-based networks require a lot of data to learn, often through extensive iterative training. When new data is encountered, the models must inefficiently relearn their parameters to adequately incorporate the new information without catastrophic interference. Architectures with augmented memory capacities, such as Neural Turing Machines (NTMs), offer the ability to quickly encode and retrieve new information, and hence can potentially obviate the downsides of conventional models. Here, we demonstrate the ability of a memory-augmented neural network to rapidly assimilate new data, and leverage this data to make accurate predictions after only a few samples. We also introduce a new method for accessing an external memory that focuses on memory content, unlike previous methods that additionally use memory location-based focusing mechanisms.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

goolulusaurs
Neutral mention · rule
The Neural Turing Machine has been expanded upon considerably since that paper, and inspired a number of other memory augmented neural network architectures, including: https://www.gwern.net/docs/rl/2016-graves.pdf , https://arxiv.org/abs/1605.06065 and https://arxiv.org/abs/1803.10760 . If you don't want to read a bunch of papers this video also has a good discussion of that last paper, describing how a memory system based on the NTM has been applied to reinforcement learning to achieve very impressive results that seem to me to be a very significant step towards human-like general purpose intelligence: https://www.youtube.com/watch?v=9z3_tJAu7MQ
Original comment Deep learning: a critical appraisal
nl
Neutral mention · rule
One shot learning is such an active area of research there's a long Wikipedia page[1] about it. I think the SOTA is probably [2], which came out of DeepMind. There's still a way to go before it matches ResNet performance on ImageNet (or even human performance on any real task) though. [1] https://en.wikipedia.org/wiki/One-shot_learning [2] https://arxiv.org/abs/1605.06065
Original comment Open-sourcing DeepMind Lab
vonnik
Neutral mention · rule
[Disclosure: I work for a deep-learning company.] Robin's post reveals a couple fundamental misunderstandings. While he may be correct that, for now, many small firms should apply linear regression rather than deep learning to their limited datasets, he is wrong in his prediction of an AI bust. If it happens, it will not be for the reasons he cites. He is skeptical that deep learning and other forms of advanced AI 1) will be applicable to smaller and smaller datasets, and that 2) they will become easier to use. And yet some great research is being done that will prove him wrong on his first point. https://arxiv.org/abs/1605.06065 https://arxiv.org/abs/1606.04080 One-shot…
Read the full comment
learning, or learning from a few examples, is a field where we're making rapid progress, which means that in the near future, we'll obtain much higher accuracy on smaller datasets. So the immense performance gains we've seen by applying deep learning to big data will someday extend to smaller data as well. Secondly, Robin is skeptical that deep learning will be a tool most firms can adopt, given the lack of specialists. For now, that talent is scarce and salaries are high. But this is a problem that job markets know how to fix. The data science academies popping up in San Francisco exist for a reason: to satisfy that demand. And to go one step further, the history of technology suggests that we find ways to wrap powerful technology in usable packages for less technical people. AI is going to be just one component that fits into a larger data stack, infusing products invisibly until we don't even think about it. And fwiw, his phrase "deep machine learning" isn't a thing. Nobody says that, because it's redundant. All deep learning is a subset of machine learning.
Original comment This AI Boom Will Also Bust
YeGoblynQueenne
Neutral mention · rule
That's a nice article in that it manages to get excitement across without forgetting to balance it somewhat. Kind of a rare thing these days. Btw, I'm interested to hear how well training with large one-hot encoded vectors scales. A paper someone pointed me to recently on HN suggested that it doesn't scale very well: One-shot Learning with Memory-Augmented Neural Networks [https://arxiv.org/abs/1605.06065]
Original comment Embed, encode, attend, predict: the new deep learning formula for NLP models
ak4g
Neutral mention · rule
This is just an outdated view of the state-of-the-art. It's understandable, given that it's outdated by maybe six months, iff you're willing to go with preprints. https://arxiv.org/abs/1605.06065 A hobbyist looking for something plug-and-play will still generally want lots of data; the cutting edge is not exactly "curl|bash"-able. But the papers coming out this year have been dispatching what I thought would be entire areas of study in a dozen pages, one after another after another. Not only do I think it's a "when" and not an "if", I think the timelines people throw around date to "ancient" times - meaning, a few years ago. Given where were are right now, what we should be asking is whether "decades" should be plural.
Original comment Show HN: DeepRhyme (D-Prime) – Generating dope rhymes with machine learning
jimfleming
Neutral mention · rule
> Only at such a high level of abstraction as to be meaningless. I'm not sure what this means or how the abstractions are meaningless? From Gabor filters to concepts like "dog", the abstractions are quite meaningful (in that they function well), even if not to us. > They are not. Hundreds of man years worth of engineering time go into each of those systems, and none of those systems generalizes to anything other than the task it was created for. That's nothing like human intelligence. This isn't strictly true if we look at the ability to generalize as a sliding scale. The level of generalization has actually increased significantly from expert systems to machine learning…
Read the full comment
to deep learning. We have not reached human levels of generalization but we are approaching. Consider that DL can identify objects, people, animals in unique photos never seen before and that more generally the success of modern machine learning is it's ability generalize from training to test time rather than hand engineering for each new case. Newer work is even able to learn from just a few examples[0] and then generalize beyond that. Or the Atari work from DeepMind that can generalize to dozens/hundreds of games. None of those networks are created specifically for Break Out or Pong. It's also not entirely fair to discount the hundreds of years of engineering considering most of these systems are trained from scratch (randomness). Humans, however, benefit from the preceding evolution which has a time scale that far exceeds any human engineering effort. :) [0] https://arxiv.org/abs/1605.06065
Original comment AI, Deep Learning, and Machine Learning: A Primer [video]
dave_sullivan
Neutral mention · rule
> If this is the state of the art in machine translation, how on Earth can you expect a machine that can converse on human level in three years? Here are 2 good papers--one from last year, one very recent--that might change your mind. https://arxiv.org/pdf/1506.05869.pdf https://arxiv.org/pdf/1605.06065v1.pdf Over the past 10 years (deep learning got going in ~2006, really), the state of the art has improved at an exponential rate, and that's not slowing down. There are plenty of reasons to be bullish. Or at least, now seems like a bad time to leave the field.
Original comment The truth about deep learning

Keep following the thread.

More in Machine learning

More from the same topics.