The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm --- an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Montezuma's Revenge, Castle Wolfenstein (not the shooter), and puzzle games in general have a problem of long term credit assignment and sparse reward. This "intrinsic reward" approach form the paper, based on pseudo counts seems to be one way to get an intermediate reward which helps the model learn toward an overall goal (winning/progress) which happens rarely. The previous best work had to pre-define the intrinsic rewards as I understand it [0], and DeepMind has been tracking this general problem for a while [1] along with a whole bunch of earlier work from the 70s/80s/90s (cited in the background of this new paper).
Credit assignment in a nutshell is "what actions…
Read the full comment
helped me get reward"? For action games this is fairly easy - there are only a few moves between rewards. For puzzlers, something like left, up, right, up, left, left, left, left, up, up could get a reward. We can see there is a cycle in there which is probably not necessary, but maybe this was a much longer path than the ideal as well. Deciding which moves should get credit is a hard problem, but an important one. [2]
If you look at the results of the original DQN paper [3] you will see the games they fared best were ones where there were frequent rewards (e.g. Breakout). Things that are puzzle-like (such as Q-Bert) fared much worse versus human benchmarks, whereas action games like Breakout (which is fully observable given 4 frame context IIRC) were generally better than the human benchmark.
This paper seems to be a big step toward deep RL for more than just short term decisions and a huge jump towards goal oriented planning.
[0] Kulkarni et. al https://arxiv.org/abs/1604.06057
[1] Mohamed, Rezende https://arxiv.org/pdf/1509.08731.pdf
[2] http://www.scholarpedia.org/article/Reinforcement_learning#....
[3] Nature results are better but paywalled :/ NIPS paper here https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf . http://www.nature.com/nature/journal/v518/n7540/abs/nature14... - Figure 3
The utility function can be specified in a way to build up something that looks like an internal motivation system. This is often referred to as intrinsically motivated reinforcement learning.
A recent paper by 2 Google Deepmind researchers on this topic:
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning
Shakir Mohamed and Danilo J. Rezende
http://arxiv.org/abs/1509.08731
and a somewhat older survey paper on the same topic:
How can we define intrinsic motivation?
Pierre-Yves Oudeyer, Frederic Kaplan
http://www.pyoudeyer.com/epirob08OudeyerKaplan.pdf
The point is that supervised training on large networks requires millions or billions of examples. In RL, that means potentially billions of execution trajectories, which can be really expensive if you're executing on an actual, physical robot. And even then, supervised classification objectives are more informative than just a scalar reward: when a supervised system makes a wrong prediction you don't just tell it "sorry, that's wrong", you also tell it what the right prediction would have been. That extra information is really valuable for learning.
In the black-box RL setting, you only see whether what you did was right or wrong, not what the right thing would have been…
Read the full comment
to do. And unlikely a classification system where the output space is relatively small (ImageNet has 1000 classes), an RL agent is searching over an exponentially large space of possible trajectories. Which means that without some additional source of supervision you can spend a long time wandering in the wilderness with no idea of whether what you're doing is reasonable or how to get any reward at all. And when you do get some reward, you have no idea which of the possibly hundreds or thousands of actions you took deserves the credit.
A lot of recent RL research is about finding additional sources of supervision, such as training an agent to mimic the policy of an "expert" (e.g., a search algorithm that runs in a simulator to find an optimal solution, but which requires too much computation to actually apply directly at test time, as in http://arxiv.org/abs/1504.00702), or coming up with proxy objectives like "empowerment" or "curiosity" (which you can define in terms of information theoretic quantities, such as mutual information between the agents' action sequence and future state, e.g. http://arxiv.org/abs/1509.08731) to supplement the actual reward signal. This latter path, the notion of "intrinsic reward" that paulsutter alluded to, is in some sense the merger of RL with unsupervised learning, and a lot of the power is that you're getting new reward signals constantly, not just when you finally manage to achieve some arbitrarily difficult task.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.