Skip to content
The methodOpen data
Back to the collection

On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models

Juergen Schmidhuber

2015 · arXiv

Machine learningAlgorithmsStatistics & probability
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

This paper addresses the general problem of reinforcement learning (RL) in partially observable environments. In 2013, our large RL recurrent neural networks (RNNs) learned from scratch to drive simulated cars from high-dimensional video input. However, real brains are more powerful in many ways. In particular, they learn a predictive model of their initially unknown environment, and somehow use it for abstract (e.g., hierarchical) planning and reasoning. Guided by algorithmic information theory, we describe RNN-based AIs (RNNAIs) designed to do the same. Such an RNNAI can be trained on never-ending sequences of tasks, some of them provided by the user, others invented by the RNNAI itself in a curious, playful fashion, to improve its RNN-based world model. Unlike our previous model-building RNN-based RL machines dating back to 1990, the RNNAI learns to actively query its model for abstract reasoning and planning and decision making, essentially "learning to think." The basic ideas of this report can be applied to many other cases where one RNN-like system exploits the algorithmic information content of another. They are taken from a grant proposal submitted in Fall 2014, and also explain concepts such as "mirror neurons." Experimental results will be described in separate papers.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

gwern
Neutral mention · rule
This is an old idea dating back to the earliest algorithmic information theory work in the '50s/60s like Ray Solomonoff on universal induction: given a powerful learning algorithm, create AI by encoding all possible data as a binary stream (with no further preprocessing or engineering), and simply predict each successive bit. This leads to AIXI, the compression-as-intelligence paradigm (and online learning / meta-learning), and most recently, the success of self-supervised learning & generative modeling. See for example Burfoot https://arxiv.org/abs/1104.5466 or Schmidhuber https://arxiv.org/abs/1511.09249#schmidhuber + https://arxiv.org/abs/1802.08864#schmidhuber (among many others) or (if I may shill my own work) https://gwern.net/aunn
Original comment Ask HN: If we train an LLM with “data” instead of “language” tokens
hardmaru
Neutral mention · rule
> How did you get into contact with Schmidhuber for co-authoring? What stage was the research at when he joined? The first time I discussed this topic with Jürgen Schmidhuber was at NIPS 2016, when he gave a talk about "Learning to Think" [1], during the break at one of the sessions, and we kept in contact afterwards. > Were you expecting the net to generalize from dream to reality, before you wrote the paper, or did this materialize during experimentation? When I tried this, I didn't expect this to work at all, to be honest! And in fact, as discussed in the paper, it didn't work at the beginning (the agent would just cheat the world model). That's why I tried to adjust…
Read the full comment
the temperature parameter to control the stochasticity of the generated environment, and trained the agent inside a more difficult dream. > Do you expect this approach is also feasible for more difficult games: higher dimensionality, longer delayed rewards? I expect the iterative training approach to be promising for difficult games with higher dimensionality, where we need to use better V and M models with more capabilities and capacities (we can already find many candidates for V/M already by looking at the deep learning literature), and still train these models efficiently with backprop on GPUs/TPUs. Using policy search methods such as evolution (or even augmented random search), allow us to work only with cumulative rewards we see at the end, rather than demanding a dense reward signal at every single time step, and I think this will help cope with environments with sparse, delayed rewards. Even in the experiments in this paper, we only work with cumulative rewards at the end of each rollout, and we don't care about intermediate rewards. > Both congrats and thanks for writing this very accessible paper. Really found this a creative paper with a lot of inspiration, and the presentation of the results was marvelous. (BTW: I remember you from the RNN-volleyball game. Back then you had quite some jealous detractors, telling you DeepMind would be too difficult/academic for you. You sure shut those people up!) Thanks! The RNN-volleyball game from 2015 was a lot of fun to make. Back then, I trained the agents using self-play, with evolution, and I remember people telling me I should really be using DQN or something back then. Fast forward a few years, self-play is now a really popular area of research (for instance, many nice works from OpenAI and DeepMind last year), and evolution methods are really making a comeback. I think it is best to work with something you believe in, and sometimes it is okay to not pursue what everyone else is doing. [1] On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models https://arxiv.org/abs/1511.09249
Original comment A step-by-step guide to the “World Models” AI paper
nrmn
Neutral mention · rule
I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then: Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850 Older one, but important to understand deeply since other recent ideas have come from this! Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks - http://arxiv.org/abs/1511.06434 Unitary Evolution Recurrent Neural Networks - http://arxiv.org/abs/1511.06464 State of the Art Control of Atari Games Using Shallow Reinforcement Learning - http://arxiv.org/abs/1512.01563 Interesting discussion in section 6.1 on the…
Read the full comment
shortcomings/issues of DQN done by Deepmind Spectral Representations for Convolutional Neural Networks - http://arxiv.org/abs/1506.03767 Deep Residual Learning for Image Recognition - http://arxiv.org/abs/1512.03385 Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) - http://arxiv.org/abs/1511.07289 I wish they did more comparisons between similar network architecture with only the units swapped out. Eg. Alexnet, Relu vs Alexnet, Elu. On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models - http://arxiv.org/abs/1511.09249 Just a few from my list :)
Original comment Ask HN: Papers you read in 2015?

Keep following the thread.

More in Machine learning

More from the same topics.