We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
The rules of Go are not exactly "programmed in" to AlphaGo; they are only provided so the model knows what the valid moves and win conditions are as it self-trains. The latest versions of the model can be quickly adapted to play other games[0] and MuZero can "master games without knowing their rules"[1]. Also, some of DeepMind's earliest work was a deep reinforcement learning model that could play seven different Atari games with no adjustments needed to switch games[2].
[0] https://en.wikipedia.org/wiki/AlphaGo_Zero
[1] https://en.wikipedia.org/wiki/MuZero
[2] https://arxiv.org/abs/1312.5602
PLAGIARISM
Third paragraph of Intro in original DeepMind paper https://arxiv.org/pdf/1312.5602.pdf
&
Second paragraph of Intro(5th line) of this project.
http://nihit.github.io/resources/spaceinvaders.pdf
They didn't even bother to do something original except for using the RAM state instead of pixel values like the original paper. It's something that's easily doable with the openAI gym. This seems like a course project at stanford.
Looks really cool.
I recently hit a roadblock when trying to implement the original DeepMind Atari algorithm [0] with TensorFlow. They don't mention this in the paper, but the network wasn't trained to convergence at each training step (maybe this would be obvious to people more well-versed in deep learning, but it wasn't to me coming from a classical RL background).
As it turns out, TensorFlow's optimizers don't have a way to manually terminate training before convergence. That meant I was getting through several orders of magnitude fewer training steps than the DeepMind team did, even when accounting for my inferior hardware. This might not be a problem in some learning…
Read the full comment
cases, where training more on certain examples lets you extract more information from them, but in games with sparse rewards it's bad.
Of course, TensorFlow does let you do the gradient calculations and updates by hand, but I wasn't prepared to go that far at the time. Maybe in the next few weeks I'll dive back into it.
[0] https://arxiv.org/pdf/1312.5602.pdf
> once we have a comprehensive AI program that can play many kinds of games
Before AlphaGo, DeepMind released a reinforcement learning algorithm that could play many Atari games just from the raw pixels on the screen, in many games surpassing humans. The same algorithm.
https://arxiv.org/pdf/1312.5602.pdf
Reinforcement learning is a general framework for learning behavior from acting in an environment with the purpose to maximize a reward. It can be used, and was used, in multiple domains. AlphaGo used RL as well.
Saying that AlphaGo is limited because it only knows to play one game, is like saying that humans are limited because Lee Sedol could only master at world level…
Read the full comment
one game. In fact, if the software was set to learn more games, it could learn them in addition to Go.
Also, regarding other tasks: a neural net that recognizes cats can be easily made to recognize dogs too. A program that translates English to French can be made to translate other languages too. We limit software to specific domains only on account of efficiency, not because algorithms are fundamentally limited.
Recently there has been a paper "Learning without forgetting" (http://arxiv.org/abs/1606.09282v2) that underlines this very ability to span multiple domains and adapt easily to unseen tasks and data.
> Developing a program that can generate solutions for a single specific niche of problems (while impressive) is not a convincing demonstration of intelligence.
Saying that people can do many tasks is not exactly right because a particular person can only do a few tasks, those tasks she was trained to do. If I never learned German, I don't know German. That doesn't mean the brain itself is limited. AlphaGo was only trained on Go, and its internal architecture was optimized for this one task in order to make it more efficient, but the method is general and reusable. DeepMind said so themselves, the breakthrough is not that they beat Lee Sedol, but that they used a general method that can be used to do other tasks as well. It is not a limitation of AI that we generally make systems that are good at only one thing.
If there is a limitation in AlphaGo, it is that it mastered a game where the whole situation is perfectly known (the Go board), while in reality many tasks are only partially known (such as card games, for example) so there is an extra uncertainty. But DeepMind and other researchers are working on that too.
I find it really interesting that Goal 4 is a game playing agent. Deepmind has been focussing on this since the beginning[0] and actually has great progress as far as Atari games go[1].
And DeepMind is able to use a single agent rather than a different specific one for each game. I wonder if OpenAI wants to go in a different direction although RL has considerable success. Whereas the other goals definitely need a lot of work before they are "real-world" functional, especially Goal 3. But of course that depends on their definition of 'useful'.
[0]:https://www.youtube.com/watch?v=rbsqaJwpu6A
[1]:http://arxiv.org/pdf/1312.5602v1.pdf
The examples where this happens have always seemed fairly weak to me. How many of the grave errors, not just where it's the wrong type of animal or container but actually thinking it's radically different, survive an application of Gaussian blur? Furthermore self-driving cars are a combination of signals; you are going to need to simultaneously fool both LIDAR and cameras.
On top of that you are going need to fool them over multiple frames, while the sensors get a different angle on the subject as the car moves. For example in the first Deep Q-learning paper, "Playing Atari with Deep Reinforcement Learning"[0], they use four frames in sequence. That was at the end of…
Read the full comment
2013.
I don't think anyone will be able to come up with a serious example that fools multiple sensors over multiple frame as the sensors are moving. Even if they do then inducing an unnecessary emergency stopping situation is still not the same as getting the car to drive into a group of people. Even if fooled in some circumstances the cars will still be safer than most human drivers and still have a massive utilitarian moral case in relation to human deaths, on top of the economic case, to be used.
The fooling of networks is still an interesting thing, but it's been overplayed to my mind and is not particularly more interesting than someone being fooled for a split second into thinking a hat stand with a coat and hat on it is a person when they first see it out of the corner of their eye.
[0]http://arxiv.org/pdf/1312.5602.pdf page 5
Raw pixels. And the score. The score was separate, or rather a signal representing increased score. Relevant quote from the paper:
"The emulator’s internal state is not observed by the agent; instead it observes an image xt ∈ Rd from the emulator,
which is a vector of raw pixel values representing the current screen. In addition it receives a reward rt representing the change in game score."
The paper:
http://arxiv.org/abs/1312.5602
Currently, it sees a board of 3x3 states, as in:
_ _ G
G B B
_ _ G
A next step would be to use real computer vision over a 3D camera in the environment aligned to R2D@ and a deep neural network to interpret pixels instead of states. DeepMind has a paper describing this approach: http://arxiv.org/pdf/1312.5602.pdf
The speech recognition and synthesis are primitive compared to any smartphone assistant these days. The natural language processing is about equivalent to SHRDLU from the 1960s [1]. It turns out that this approach based on manually constructing syntax trees and applying simple logic can make some fun demos but is ultimately a dead end in terms of making systems that are actually useful, as was discovered in the "AI winter".
The part that controls Mario looks similar (if not identical) to this: http://aigamedev.com/open/interviews/mario-ai/
If you want to see the state of the art in using AI to play video games, look no further than "Playing Atari with Deep Reinforcement…
Read the full comment
Learning" [2], where a single general AI system learns to play many different games. The generality is what makes it impressive. Its only inputs are pixels and score, and its only outputs are joystick and button state, just like a human player. This makes it unlike these Mario systems which are hand programmed very specifically for Mario, and use special instrumentation of the game state that skips pixels entirely.
[1] http://en.wikipedia.org/wiki/SHRDLU
[2] http://arxiv.org/abs/1312.5602
While this complaint generally has validity, their paper [1] does IMO present an advance; it's not just handing a bunch of labeled data off to a large neural network.
IIRC (forgive me, I read the paper a few weeks ago) the solution is at its core a reinforcement learning system, with the deep net only making up the component that predicts reward from a (state, action) pair. With that in hand, there remains the non-trivial RL problem of balancing "exploration vs exploitation" in learning good strategies to play the game(s). While NN's have been used in this capacity before, I believe that, as other comments have mentioned, using a deep net to learn to map a high-dimension…
Read the full comment
state-action space (e.g,the state of the game represented as pixels of the screen at a particular time) to expected reward in real time was indeed an advance, both theoretical and technical.
And, oh yeah, I just remembered that a University of Texas research group is doing work in this area too (there was a recent paper [2] from Peter Stone and others).
(Edited for clarity)
(Edited again to suggest another paper).
[1] - http://arxiv.org/pdf/1312.5602.pdf
[2] - http://www.cs.utexas.edu/~pstone/Papers/bib2html-links/TCIAI...
Here's the original research paper if you're interested.
http://arxiv.org/abs/1312.5602
I'll just quote their introduction instead of trying to summarize the paper:
"Our goal is to create a single neural network agent that is able to successfully learn to play as many of the games as possible. The network was not provided with any game-specific information or hand-designed visual features, and was not privy to the internal state of the emulator; it learned from nothing but the video input, the reward and terminal signals, and the set of possible actions—just as a human player would. Furthermore the network architecture and all hyperparameters used for training were kept constant across the games. So far the network has outperformed all previous RL algorithms on six of the seven games we have attempted and surpassed an expert human player on three of them."
A note -- if you're linking to arXiv, it's better to link to the abstract (http://arxiv.org/abs/1312.5602) rather than directly to the PDF. From the abstract, one can easily click through to the PDF; not so the reverse. And the abstract allows one to do things like see different versions of the paper, search for other things by the same authors, etc.
I like this one because it is easier to get an idea of what the layout of the neural network itself is, and gives more control.
Off the topic of neural networks in javascript but on the topic of neural networks in general, the readme links a cool paper on applying convolutional neural networks to old games.
http://arxiv.org/abs/1312.5602
Exactly. Interesting, Yann LeCun in his G+ post (https://plus.google.com/u/0/+YannLeCunPhD/posts/LnKBPi9cCWm) explicitly mentioned that the acquired ML scientists have been focused in reinforcement learning. Also, they have published ArXiv paper on the subject: http://arxiv.org/pdf/1312.5602v1.pdf.
I remember that they had an impressive demo at the NIPS Deep Learning Workshop in December (probably a highlight for me and a few people I talked to afterwards) of a reinforcement learning agent playing Atari 2600 Games where the input consisted of an image that was fed through a Convolutional Neural Network. There is a paper on Arxiv that describes the approach ("Playing Atari with Deep Reinforcement Learning"[0]) but unfortunately I don't think they made the videos available.
Deep Mind has a whole bunch of talented and serious people so this is an exciting acquisition.
[0] http://arxiv.org/pdf/1312.5602v1.pdf
It's worth pointing out Google just bought the startup with the guys who wrote the "Playing Atari with Deep Reinforcement Learning" paper which is supposedly just an algorithm working on the raw pixels and then beating all other programs and half of the human experts.[1]
[1] http://arxiv.org/pdf/1312.5602v1.pdf
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.