Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski
Tom Schaul, David Saxton, Remi Munos
2016 · arXiv
We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
5 selected excerpts · 5 total mentions
There has been a fair bit of past work exploring the idea you described (examples: https://arxiv.org/pdf/1606.01868.pdf, https://arxiv.org/pdf/1703.01310.pdf, https://pathak22.github.io/noreward-rl/resources/icml17.pdf, https://openreview.net/forum?id=H1RPJf5Tz). Such methods can't solve games like Montezuma's revenge to a comparable level of performance yet, but I'm sure they'll eventually get there.
By "make" you meant "match some sort of a simple function approximation after hardcoding lots of knowledge about the system and the general function" right? Which is essentially what the neural networks and all the other optimization algorithms were made for? (The algorithm as described will require a huge database for a game that is even slightly more complicated than Infinite Mario. And we don't even have the sources to try that.) Even the object motion tracker part will choke in 3D environment. (It is a greedy matcher as they described it.) Speaking of impressed, Google DeepMind paper is way more actually feasible to improve upon and rich in detail: https://arxiv.org/pdf/1606.01868v1.pdf Compare the two papers in straight quality. I understand why you'd publish any worthless junk in the current academic culture and do not agree we should actually do it.
Thanks. We've changed the URL from http://www.wired.co.uk/article/google-ai-montezuma-revenge. At first to https://arxiv.org/abs/1606.01868, but since everyone's going to want to watch the video, that seems a bit too abstruse. Hopefully someone will say meaningful things about the paper too.
And here's the related paper, "Unifying Count-Based Exploration and Intrinsic Motivation": https://arxiv.org/abs/1606.01868 Tech journalism isn't at its best when it can't distinguish between WOPR and a team of researchers.
If you end up blocked by a popup blocker, or just don't feel like reading fluff, or like me you've got the browser locked down too tightly for their integrated video player to work, the paper is at https://arxiv.org/pdf/1606.01868v1.pdf and the video mentioned is (probably, since I didn't see the original) at https://www.youtube.com/watch?v=0yI2wJ6F8r0 .
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.