Yan Duan, John Schulman, Xi Chen
Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel
2016 · arXiv
Deep reinforcement learning (deep RL) has been successful in learning sophisticated behaviors automatically; however, the learning process requires a huge number of trials. In contrast, animals can learn new tasks in just a few trials, benefiting from their prior knowledge about the world. This paper seeks to bridge this gap. Rather than designing a "fast" reinforcement learning algorithm, we propose to represent it as a recurrent neural network (RNN) and learn it from data. In our proposed method, RL$^2$, the algorithm is encoded in the weights of the RNN, which are learned slowly through a general-purpose ("slow") RL algorithm. The RNN receives all information a typical RL algorithm would receive, including observations, actions, rewards, and termination flags; and it retains its state across episodes in a given Markov Decision Process (MDP). The activations of the RNN store the state of the "fast" RL algorithm on the current (previously unseen) MDP. We evaluate RL$^2$ experimentally on both small-scale and large-scale problems. On the small-scale side, we train it to solve randomly generated multi-arm bandit problems and finite MDPs. After RL$^2$ is trained, its performance on new MDPs is close to human-designed algorithms with optimality guarantees. On the large-scale side, we test RL$^2$ on a vision-based navigation task and show that it scales up to high-dimensional problems.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
9 selected excerpts · 9 total mentions
Hyperparameter optimization (including architectures) is not really meta-learning. Meta-learning, also known as "learning to learn", is more like MAML[0], RL2[1], L2RL[2], etc. 0. https://arxiv.org/abs/1703.03400 1. https://arxiv.org/abs/1611.02779 2. https://arxiv.org/abs/1611.05763
Unfortunately, many of your assertions are inaccurate. As far as game playing goes, the ways in which AI agents beat human opponents is simply a matter of computers being able to process more information in a given period of time. The selection and evaluation criteria are actually pretty similar to human decision-making, if a bit more systematic. I can expand on this point in another comment, if you'd like. Chess and Go are not solved games, and will likely not be for some time (10+ years, for Go at least). Solving non-deterministic problems are even harder, and many likely will not be accomplished within our lifetimes[1]. There is no AI system that can generally make…
more accurate diagnoses than a physician. We're not there quite yet, and it will be a while. The best we've come are some pretty advanced expert systems, but these require very heavy input from physicians[2]. Please provide sources to validate your claims on the progress of AI. Extrapolation from data is dangerous, extrapolation from falsity is ignorance. As far as your take on understanding goes, the human "meaning" of things is inherently subjective. If we define "meaning" as some thing's value or role based on environmental context, then just like humans, any artificially intelligent system will only be able to determine meaning based on observations of the environment and both individual and collective experience. It's interesting that you seem to be focusing primarily on social domains that have inherently "human" contexts. Beyond misunderstanding the point of this paper, and the way AI systems work, I think you're missing the point. AI is, and for the time being will remain, an extension of the human mind. The decision-making needs to be developed (at some core level) by a human. Those goals need to be set by a human. The experiences and observational capabilities ultimately need to be determined by a human. Even AI systems that build other AI systems need to be directed to do so, and with strict goals, set by a human [3]. I highly suggest you take a moment to read openai's mission statement: https://openai.com/about/. AI is a tool. And like any powerful tool it must be used responsibly, freely, and openly. Openai is pursuing this goal and making efforts to ensure that this tool is available to as many people as possible to avoid the abuses implicit to your concerns. You obviously have an interest in AI and some knowledge of the field, but I worry your comments veer a bit towards fear-mongering. I suggest you use openai and resources like it to enrich your knowledge of both the advances and concerns of AI, because those are important, and we definitely need people thinking about these things. Ultimately, you are absolutely correct that eventually these systems will probably have the technical capability to influence elections, the economy, and more. But the only way they will is under the direction of humans. It it not the machine you should fear, but the man behind it. The same thing you ought to have been fearing all along. 1. http://fragrieu.free.fr/SearchingForSolutions.pdf 2. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1307157/ 3. https://arxiv.org/abs/1611.02779
Handling delays (and the uncertainty they entail) is a huge challenge, and I think it'll be a rich area of research. The simplest part of the problem is that delays in action or perception also slow the propagation of reward signals, and credit assignment is still a really hard problem. Thinking further afield, future models could learn to adapt their expectations to fit the behavior of a particular opponent. This kind of metalearning is pretty much a wide open problem, though a pair of (roughly equivalent) papers in this direction recently came out from DeepMind: https://arxiv.org/abs/1611.05763 and OpenAI: https://arxiv.org/abs/1611.02779 It's going to be really exciting to see how these techniques scale.
Note: The original title noted OpenAI's role in the paper, which maybe seen by loading the PDF and reading the author credits. https://arxiv.org/pdf/1611.02779v2.pdf
Other equally exciting papers that relates to learning to learn in DL. "Neural Architecture Search with Reinforcement Learning" https://arxiv.org/abs/1611.01578 "RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning" https://arxiv.org/abs/1611.02779 "Designing Neural Network Architectures using Reinforcement Learning" https://arxiv.org/abs/1611.02167
Meta- reinforcement learning could prove to be such breakthrough, see [1],[2]. Also next generation ASIC accelerators (Google's TPU, Nervana) can give 10x increase in NN performance over a GPU manufactured on the same process, with another 10x possible with some form of binarized weights, e.g. BNN, XNOR-net. There are also interesting techniques to update the model's parameters in a sparse manner. So, there certainly is a lot of room left for performance improvements! 1. https://arxiv.org/abs/1611.05763 2. https://arxiv.org/abs/1611.02779
On Google and Oxford's cases, I don't think it is common to broadcast among your colleagues in different groups your proposed ideas before you actually go into it. So I think it may happen even in the same company/university. For the meta-learning papers, you may have interests to read the related work part of the RL^2 paper https://arxiv.org/pdf/1611.02779.pdf. Quoted as follows, "Our work draws inspiration from a particular line of work (Younger et al., 2001; Santoro et al., 2016; Vinyals et al., 2016), which formulates meta-learning as an optimization problem, and can thus be optimized end-to-end via gradient descent." "Another line of work (Hochreiter et al.,…
2001; Younger et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2016) studies meta-learning over the optimization process. There, the meta-learner makes explicit updates to a parametrized model." Inspired by the same works, apply the meta learning idea into RL problems, meet the ICLR deadline together. Still make sense right?
Even more interesting - an OpenAI paper on the roughly same subject/technique came out a little earlier (RL2 fast reinforcement learning for slow reinforcement learning https://arxiv.org/pdf/1611.02779.pdf). I think parallel inventions tends to indicate that multiple people have stumbled on a similar good idea at the same time. (disclaimer: I work at OpenAI. v pleased to see these two papers emerge so close together)
Looks like the "UNREAL" (https://arxiv.org/abs/1611.05397), "Learning to reinforcement learn" (https://arxiv.org/abs/1611.05763) and "RL^2" (https://arxiv.org/abs/1611.02779) are state of art in pure RL for now. Finally there is a trend of using recurrent neural network as a top component of the Q-network. Perhaps we will see even more sophisticated RNNs like DNC and Recurrent Entity Networks applied here. Also we'll see meta-reinforcement learning applied to a curriculum of environments.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.