We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
No need stupid moving texts.
CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/
The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/
Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/
ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436...
Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385
Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122
Identity Mappings in Deep Residual Networks -…
Read the full comment
https://arxiv.org/abs/1603.05027
Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595
Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391
Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473
Pointer Networks - https://arxiv.org/abs/1506.03134
Attention Is All You Need - https://arxiv.org/abs/1706.03762
The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/
Neural Turing Machines - https://arxiv.org/abs/1410.5401
A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427
Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822
Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212
Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361
GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965
Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf
A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077
The First Law of Complexodynamics - https://scottaaronson.blog/?p=762
Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903
Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X
Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731
Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
This is quite a good overview, and parts reflect well how things played out in language model research. It's certainly true that language models and deep learning were not considered particularly promising in NLP, which frustrated me greatly at the time since I knew otherwise!
However the article misses the first two LLMs entirely.
Radford cited CoVE, ELMo, and ULMFiT as the inspirations for GPT. ULMFiT (my paper with Sebastian Ruder) was the only one which actually fine-tuned the full language model for downstream tasks. https://thundergolfer.com/blog/the-first-llm
ULMFiT also pioneered the 3-stage approach of fine-tuning the language model using a causal LM objective…
Read the full comment
and then fine-tuning that with a classification objective, which much later was used in GPT 3.5 instruct, and today is used pretty much everywhere.
The other major oversight in the article is that Dai and Le (2015) is missing -- that pre-dated even ULMFiT in fine-tuning a language model for downstream tasks, but they missed the key insight that a general purpose pretrained model using a large corpus was the critical first step.
It's also missing a key piece of the puzzle regarding attention and transformers: the memory networks paper recently had its 10th birthday and there's a nice writeup of its history here: https://x.com/tesatory/status/1911150652556026328?s=46
It came out about the same time as the Neural Turing Machines paper (https://arxiv.org/abs/1410.5401), covering similar territory -- both pioneered the idea of combining attention and memory in ways later incorporated into transformers.
You seem to be talking about neural Turing machines: https://arxiv.org/abs/1410.5401
Combining these with LLM sounds indeed quite interesting, I don't know why they haven't been used much.
There's no neural network you could train using supervised learning to solve sorting in the general case.
To train a neural network, the training set and test set need to come from a common probability distribution but the inputs of sorting algorithms can be random. There's no relation between what you could possibly train your network on and what you'd ask it to do.
Put another way - the input and output of your model would have zero correlation. There's nothing to train the model to pick up on.
Interesting related work: Graves[0] trained a "neural Turing machine" to encode instructions for a basic sorting algorithm.
[0] https://arxiv.org/abs/1410.5401
Are you aware of the Neural Turing Machine (https://arxiv.org/abs/1410.5401) or Differentiable Neural Computer (https://deepmind.com/blog/differentiable-neural-computers/)?
They're not exactly what you describe, which I suppose is a truly online model that knows how/when to update its own parameters.
But those two models do incorporate a similar concept of external memory, whereby the controller is trained via BP to read/write to a tensor (essentially a form of soft-addressable memory available at inference time).
As far as I recall, these were never applied beyond toy problems, and it seems this line of research hasn't been very active (at least since the Transformer's "memorize all the things" approach started performing exceptionally well on all the benchmarks). I haven't read the paper you linked just yet - it may well be relevant.
"We extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes. The combined system is analogous to a Turing Machine or Von Neumann architecture but is differentiable end-to-end, allowing it to be efficiently trained with gradient descent. Preliminary results demonstrate that Neural Turing Machines can infer simple algorithms such as copying, sorting, and associative recall from input and output examples." https://arxiv.org/pdf/1410.5401.pdf
While I haven't read those particular papers yet, one common pattern in the ML literature seems to be:
1) Identify a commonly seen problem with deep learning architectures, whether that's large data volumes, lack of transfer learning, etc.
2) Invent a solution to the problem.
3) Test that the solution works on toy examples, like MNIST, simple block worlds, simulated data, etc.
4) Hint that the technique, now proven to work, will naturally be extended to real data sets very soon, so we should consider the problem basically solved now. Hooray!
5) Return to step #1. If anyone applies the technique to real data sets, they find, of course, that it doesn't generalize well and…
Read the full comment
works only on toy examples.
This is simply another form of what happened in the 60s and 70s, when many expected that SHRDLU and ELIZA would rapidly be extended to human-like, general-purpose intelligences, with just a bit of tweaking and a bit more computing power. Of course, that never happened. We still don't have that today, and when we do, I'm sure the architecture will look very very different from 1970s AI (or modern chatbots, for that matter, which are mostly built the same way as ELIZA).
I don't mean to be too cynical. Like I said, I haven't read those particular papers yet, so I can't fairly pass judgement on them. I'm just saying that historically, saying problem X "has been addressed" by Y doesn't always mean very much. See also eg. the classic paper "Artificial Intelligence Meets Natural Stupidity": https://dl.acm.org/citation.cfm?id=1045340.
EDIT: To be clear, I'm not saying that people shouldn't explore new architectures, test new ideas, or write up papers about them, even if they haven't been proven to work yet. That's part of what research is. The problem comes when there's an expectation that an idea about how to solve the problem means that the problem is close to being solved. Most ideas don't work very well and have to be abandoned later. Eg., as one example, this is the Neural Turing Machine paper from a few years back:
https://arxiv.org/pdf/1410.5401.pdf
It's a cool idea. I'm glad someone tried it out. But the paper was widely advertised in the mainstream press as being successful, even though it was not tested on "hard" data sets, and (to the best of my knowledge) it still hasn't several years later. That creates unrealistic expectations.
Input and output sizes don't have to be fixed. E.g. speech recognition doesn't work with fixed sized inputs. Natural language processing deals with many different length sequences. seq2seq networks are explicitly designed to deal with problems that have variable length inputs and outputs that are also variable in length and different from the input.
How would you sort integers? using neural turing machines: https://arxiv.org/abs/1410.5401
NMTs and other memory network architectures have explicit memory as state (including stacks!), indeed any recurrent neural net has state.
Are NNs Turing complete? Yes! http://binds.cs.umass.edu/papers/1992_Siegelmann_COLT.pdf
Neural Turing Machines by Alex Graves, Greg Wayne, Ivo Danihelka
https://arxiv.org/abs/1410.5401
A Neural Turing machine is a type of neural network with addressable memory - it learns rather than being explicity programmed.
The net also learns to boot, i.e. how to read/ write it's own memory from scratch.
Stochastic Gradient Descent learning relies on backpropagation of the error corrections to the individual weights to make a slight improvement - this requires a differentiable neural net.
This version innovates with content addressable memory. The authors demonstrate learning from big data sets and with reinforcement learning, (i.e. trial and error as an embodied agent).
yequalsx: yes, almost by definition, memory made up of bits should be modeled with a field of characteristic 2. That seems the most "natural" way to do it.
However, the authors of this paper are thinking about "computers" in a broader sense.
Take a look at these two papers:
* "Neural Turing Machines:" https://arxiv.org/abs/1410.5401
* "Hybrid computing using a neural network with dynamic external memory:" http://www.nature.com/nature/journal/v538/n7626/full/nature2... and a related blog post at https://deepmind.com/blog/differentiable-neural-computers
The memory space of these "neural" computers is formally modeled as R^n (n being the total number of parameters and memory…
Read the full comment
addresses), and implemented approximately with 32-bit floating point numbers in practice (i.e., with a memory space of (2^32)^n bits). As you read the paper linked to by the OP, think about these neural computers instead of the one atop your desk.
EDITS: removed sloppy math and lightly edited some sentences.
The author proposes a lot of vague ideas in this article (for example "I believe one of the biggest problems is the use of Error Propagation and Gradient Descent") without references or any solid proofs why they are necessary to solve the proposed program (Automate programming using ML?).
In fact there is already a lot of solid work just on this subject:
* Learning algorithms from examples
http://arxiv.org/abs/1511.07275
https://arxiv.org/abs/1410.5401
* Generating source code from natural language description
http://arxiv.org/abs/1510.07211
* And, the most closest work to what author probably wants, a way to write a program in forth while leaving some functions as neural…
Read the full comment
blackboxes to be learned from examples:
http://arxiv.org/abs/1605.06640
* Also there is a whole research program by nothing less than Facebook AI Research that explicitly aims at creating a conversational AI agent that is able to translate user's natural language orders into programs (asking to the user additional questions if necessary): http://arxiv.org/abs/1511.08130 (there is also a summary here http://colinraffel.com/wiki/a_roadmap_towards_machine_intell... )
And deepmind is also working on conversational agents: https://youtu.be/vQXAsdMa_8A?t=1265
Given current success of such models, automating simple programming tasks maybe not as much research as engineering and scaling up problem.
There is a lot of exciting machine learning research out there nowadays. Almost all of this research is available for free from papers posted on arxiv. It is a really good idea to read more about state of the art before coming with new ideas.
You might enjoy this paper if you haven't already read it. http://arxiv.org/abs/1410.5401
Jump to the experiments section to get an overview of what it can do.
Here's the original paper on Neural Turing machines: https://arxiv.org/abs/1410.5401
IIRC, that paper's results focus on the model's ability to extrapolate prediction of sequential data to arbitrary lengths having only been trained on sequences of a fixed length.
FYI, RNNs are a form of Deep Learning these days (in particular LSTM and GRU RNNs, very popular for language translation and much more for a few years now). There is also research lately into weirder types of neural nets such as 'Neural Turing Machines' (http://arxiv.org/abs/1410.5401).
One related work that came to mind was Neural Turing Machines, which augments the LSTM network with a (fixed size) memory [1]. I did see a brief reference to the Graves et al. paper in the paper linked to in the original post, but I'm also wondering if someone more knowledgeable could elaborate on the difference.
My own guess would be that the models are similar (both using NNs with external memory, trained through backprop), but this has a simpler memory structure and so could be trained more easily. It seems like the choice of regular RNNs vs. an LSTM network doesn't matter much since the Graves paper also compared LSTM vs. feedforward controllers and the results were about the same, with the feedforward controllers actually learning a bit faster. The key of both works seems to be in making a continuous system that can be trained to use external memory through backprop.
[1] http://arxiv.org/abs/1410.5401
The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3].
There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15],…
Read the full comment
drawing things [16] and many more.
[1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-in...
[2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it...
[3] http://arxiv.org/abs/1403.6652
[4] http://arxiv.org/abs/1502.03240
[5] http://arxiv.org/abs/1411.4734
[6] http://arxiv.org/abs/1409.0473
[7] http://arxiv.org/abs/1502.03044
[8] http://arxiv.org/abs/1410.5401
[9] http://arxiv.org/abs/1506.02216
[10] http://arxiv.org/abs/1507.04808
[11] http://arxiv.org/abs/1503.08895
[12] http://arxiv.org/abs/1308.0850
[13] http://arxiv.org/abs/1412.3409
[14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
[15] http://www.sciencedirect.com/science/article/pii/S0893608014...
[16] https://www.youtube.com/watch?v=Zt-7MI9eKEo
> Our intelligence grows from childhood over many years of training, through our physical and mental interactions with the world. We learn methods along side experiences. It is not about the speed of linear computation, or the amount of memory.
There's nothing in principle that prevents machines from learning methods. For example, there's software that can learn some simple algorithms: http://arxiv.org/abs/1410.5401
> Do we know what intelligence is?
It's hard to pin down exactly what anything is (unless it's something we constructed from axioms). I'd be hard pressed to tell you exactly what a car is (must it have exactly 4 wheels?) -- but I still look for them when I…
Read the full comment
cross the street.
You know intelligence when you see it. Bill Gates is more intelligent than a fish. That's a useful distinction to make in the world! (And that's what words are for)
> Why would be make something to imitate our own?
It's economically valuable! That's why Facebook, Google, Microsoft, Baidu, IBM are collectively investing billions of dollars per year.
> Would an artificial system have curiosity? (perish all the Internet cats!)
From the outside, curiosity looks like a drive to explore without a definite prediction about how it might help you achieve ultimate goals.
Curiosity in that sense should automatically appear in some (theoretical) systems -- for example, Marcus Hutter's AIXI formalism (https://en.wikipedia.org/wiki/AIXI).
More concretely (but less directly akin to curiosity), algorithms that make tradeoffs between exploitation and exploration, such as solutions to the Multi-armed bandit problem (https://en.wikipedia.org/wiki/Multi-armed_bandit) could display a kind of curiosity. When faced with a number of lotto levers, with independent and unknown payout schedules, the optimal solution involves switching around between exploiting known high-payout levers and becoming "curious" and trying out new levers.
The Monte Carlo tree search (https://en.wikipedia.org/wiki/Monte_Carlo_tree_search) algorithm used to make Go AIs employs such a strategy (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.102....). The algorithm tries out certain variations of play, and then sometimes becomes "curious" as to whether certain other lines of play might be better, and switches over to deepening its investigation of them.
These are just examples of things that are somewhat functionally equivalent to curiosity from the outside -- it says nothing about the internal sensation of curiosity that humans experience. I have no idea what's necessary for that!
> Why do we think that intelligence would escape and kill us?
If there _were_ extremely intelligent software (a huge if!), it does seem ridiculous to claim that it'd escape or hurt people. Why would it be driven to do such a thing? The argument espoused by some very smart people e.g. Elon Musk, Stuart Russell (http://edge.org/conversation/the-myth-of-ai#26015), Bill Gates, Stephen Hawking, et al runs a bit like this:
Intelligent software, without being malevolent per-se, might in the pursuit of its goals incidentally do things that people don't like. (More on this line of reasoning: https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...)
Humans don't harbor a particular ill-will towards ants, but if we become interested in some (orthogonal) goal like building a basement for a house, we might along the way destroy some ant colonies. When human-level intelligence arrived and went about its business, it led to the extinction of quite a few species.
Currently it doesn't work, but saying it'll never work is pretty strong.
This kind of demo shows that deep neural networks can capture the structure of language, if not the semantics, in a very general way. And we have separate evidence that they can (in principle) capture semantic meaning and algorithmic reasoning as well, for example: http://arxiv.org/pdf/1410.5401v2.pdf (the "neural Turing machines" paper from DeepMind)
Deep nets are only loosely inspired by neurobiology. That's why LeCun calls them "convolutional nets" and not "convolutional neural nets" and prefers "nodes" over "neurons".
It is, however, possible to have a deep net produce 3D models/images: https://www.youtube.com/watch?v=QCSW4isBDL0 "Learning to Generate Chairs with Convolutional Neural Networks".
I also suspect a different part of cognition is used when humans are asked to recreate a "fire truck" than when humans are asked to classify a "fire truck" from a "car". The former seems closer to using memory ("what did the last five fire trucks I saw look like?"). A fairly recent addition to deep nets is making use of memory: http://arxiv.org/pdf/1410.5401.pdf "Neural Turing Machines". So the difference may quickly become less significant.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.