Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert
Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver
2019 · arXiv
Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the MuZero algorithm which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. MuZero learns a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the reward, the action-selection policy, and the value function. When evaluated on 57 different Atari games - the canonical video game environment for testing AI techniques, in which model-based planning approaches have historically struggled - our new algorithm achieved a new state of the art. When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
13 selected excerpts · 13 total mentions
During play, yes, obviously you need an implementation of the game to play it. But in its planning tree, no: > MuZero only masks legal actions at the root of the search tree where the environment can be queried, but does not perform any masking within the search tree. This is possible because the network rapidly learns not to predict actions that never occur in the trajectories it is trained on. https://arxiv.org/pdf/1911.08265
totally! MuZero is my favourite[1] The super cool thing about MuZero is that it learns the dynamics of the problem, i.e. you don't have to give it the rules of the game, which makes the algorithm very general. For example, DeepMind threw MuZero at video compression and found that it can reduce video sizes by 6.28% (massive for something like YouTube)[2][3]. Curious if anyone else knows examples of MuZero being deployed outside of toy examples? [1] https://arxiv.org/pdf/1911.08265.pdf [2] https://arxiv.org/pdf/2202.06626.pdf [3] https://www.deepmind.com/blog/muzeros-first-step-from-resear... (edit s/Google/DeepMind)
AI Research Residency Program (1 year) | Zug/Zurich area | Full time | On site | 100-200k CHF/annum I am launching a new AI research organisation in the Zug/Zurich area. The goal of the organisation is to make a meaningful contribution towards solving AGI. I am looking for employees number no. 1 and 2. This will ideally be someone with a history of interesting research and a clear agenda for future research. I am personally interested in LLM's, MuZero (https://arxiv.org/abs/1911.08265), Dreamer (https://arxiv.org/pdf/2301.04104v1.pdf) and DreamCoder (https://arxiv.org/abs/2006.08381), and think there are some interesting systems to be found by interpolating between these…
existing areas (however, please still apply even if your research isn't in these areas). The project will be financed by my current company which has made extensive use of deep learning for the past 4 years. You will have access to the >100 top-end GPUs owned by this company. The job will be structured as a research residency program in the first year, with the hope of transitioning successful candidates into full-time roles. In your application, please include a high-level research proposal stating what you would work on. To apply, email me at : not_a_cat at fastmail. co[m].
> Most people will both lose time with false-start approaches that are scrapped To a discerning eye this pattern looks like stronger minds implementing higher quality tree search[1], either through implementing higher quality neural search heuristics for tree pruning or via greater search depth. Given relative NP-hardness of the problems[2] we are usually dealing with at our line of work and what we know about planning in machines and animals[3], it doesn't look immediately implausible. With planning capability strong enough, one doesn't need trial and error in the real world for moderately complex problems: the plan devised through internal search is good enough to work…
on the first try. Weaker planners have to compensate by executing more rollouts and using the real world as a ground truth model. 1. https://arxiv.org/abs/1911.08265 2. https://www.microsoft.com/en-us/research/wp-content/uploads/... 3. https://onlinelibrary.wiley.com/doi/abs/10.1111/ejn.13851
This is explained in Appendix A of the paper ("Comparison to AlphaZero"): https://arxiv.org/pdf/1911.08265.pdf Basically, AlphaZero was provided with a simulator that was able to distinguish legal and illegal moves and determine which future game states would be wins or losses. This was used to generate the search tree of possible states and actions. MuZero doesn't have access to a simulator, it only has access to its direct environment. MuZero excludes actions that are immediately illegal, which solves the problem you mention in your penultimate paragraph, but it needs to learn the game's dynamics in order to determine which future moves and states are possible.
After AlphaZero, DeepMind went on to develop MuZero (https://en.wikipedia.org/wiki/MuZero, paper: https://arxiv.org/abs/1911.08265), which is able to learn to play games without being provided an explicit model of their rules. "When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules."
MuZero performs a "formal search". In many more ways than one, for example optimisation is still a search for an optimal search of parameters. But I guess you mean that it doesn't perform a tree search? Quoting from the abstract of the paper on arxiv [1]: In this work we present the MuZero algorithm which, by combining _a tree-based search_ with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, withoutany knowledge of their underlying dynamics. (My underlining) If I remember correctly, MuZero is model-free in the sense that it learns its own evaluation function and reward policy etc (also going by the abstract). But…
it retains MCTS. Indeed, it wouldn't really make sense to drop MCTS from the architecture of a system designed to play games. I mean, it would be really hard to justify discarding a component that is well known to work and work well, both from an engineering and a scientific point of view. _________________ https://arxiv.org/abs/1911.08265
I agree that is vague. But there are plenty of RL algorithms that learn with essentially no domain knowledge, e.g., MuZero[1] which doesn't even use a model. That doesn't make it AGI -- it only masters a single game at a time and we currently don't know how to transfer that knowledge to other domains. [1] https://arxiv.org/abs/1911.08265
Check out MuZero. It learns an embedding of the game's state space potentially allowing AlphaGo like dominance in more ways. https://www.youtube.com/watch?v=We20YSAJZSE https://arxiv.org/abs/1911.08265 (Actually, ML is still not very good at causal reasoning so we have sometime. I'm more excited and worried about crispr at this point; what happens when we can make people genetically super human)
Instead of an alpha-beta search, AlphaZero uses a general-purpose Monte Carlo tree search (MCTS) algorithm. Source: https://science.sciencemag.org/content/362/6419/1140.full I agree with your comment about System 2 like reasoning not being common right now. I am not an expert in the field but the closest thing I have seen to learned planning is: https://arxiv.org/pdf/1911.08265.pdf
> no AI that I know of has at its disposal a full blown model of the world it operates in This is a field called Model-based Reinforcement learning, and it's quite advanced already -- there are indeed models that have an internal state reflecting the world state. A good recent example: https://papers.nips.cc/paper/7512-recurrent-world-models-fac... > deep learning model, however much we'd like to think they do, aren't capable of doing proper causal inference in a general setting This is also addressed by recent models, somewhat. Once you have an abstract world model, searching for a high reward can be just a matter of running markovian simulation on it using high reward…
heuristics (given by a network of course), like AG does. This line is also very active right now, one example is the recent MuZero. https://arxiv.org/abs/1911.08265 Inference at its core really isn't much more than an artful curve fitting (or an artful model search if you like), and it's one of the building blocks of intelligence.
There are certainly challenges, but "incorporating a world model" has been going well recently: "Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model" https://arxiv.org/abs/1911.08265
From the preprint [1]: > In Go, MuZero slightly exceeded the performance of AlphaZero, despite using less computation per node in the search tree (16 residual blocks per evaluation in MuZero compared to 20 blocks in AlphaZero). This suggests that MuZero may be caching its computation in the search tree and using each additional application of the dynamics model to gain a deeper understanding of the position. It also strikes me as possible that just not giving the system the rules to start with might have allowed it to explore more efficient strategies. [1] https://arxiv.org/pdf/1911.08265.pdf
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.