By introducing several improvements to the AlphaZero process and architecture, we greatly accelerate self-play learning in Go, achieving a 50x reduction in computation over comparable methods. Like AlphaZero and replications such as ELF OpenGo and Leela Zero, our bot KataGo only learns from neural-net-guided Monte Carlo tree search self-play. But whereas AlphaZero required thousands of TPUs over several days and ELF required thousands of GPUs over two weeks, KataGo surpasses ELF's final model after only 19 days on fewer than 30 GPUs. Much of the speedup involves non-domain-specific improvements that might directly transfer to other problems. Further gains from domain-specific techniques reveal the remaining efficiency gap between the best methods and purely general methods such as AlphaZero. Our work is a step towards making learning in state spaces as large as Go possible without large-scale computational resources.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
From a tweet of one of the authors [1]: "KataGo was trained on Tromp-Taylor rules so we evaluate our attack using this too."
This is incorrect. According to the KataGo paper [2], KataGo is trained using modified Tromp-Taylor rules (as pointed out by [3]):
"Self play games used Tromp-Taylor rules modified to not require capturing stones within pass-aliveterritory [...] In Go, a version of Benson's algorithm [1] can prove areas safe even given unboundedly many consecutive opponent moves ("pass-alive"), enabling this minor optimization."
It would be more interesting if they beat KataGo using the rules it was trained on. You could write a bot to beat KataGo in chess, but KataGo wasn't trained on chess...
[1] https://twitter.com/ARGleave/status/1587875104578359296
[2] https://arxiv.org/pdf/1902.10565.pdf
[3] https://www.reddit.com/r/baduk/comments/yl2mpr/ai_can_beat_s...
I've been working on a Python implementation that uses Gradient Boosted Decision Trees (LightGBM/Treelite) instead of using a neural network for the value/policy models:
https://github.com/cgreer/alpha-zero-boosted
It's mostly to understand how AlphaZero&Friends work. I'm also curious about how well a GBDT could do, and if there are self-play techniques that can accelerate training.
The nice thing about a GBDT is that, unlike when using a NN, you can do thousands of value/policy lookups per second on a single core. So it should be cheaper to scale self-play and run a lot of self-play experiments (assuming the self-play learnings when using the GBDT model transfer to when…
Read the full comment
you use the more-powerful NN in these environments).
If you're curious about accelerating self-play training, check out David Wu's work (https://arxiv.org/pdf/1902.10565.pdf). He's the creator of KataGo. I implemented his "Playout Cap Randomization" technique in my implementation above and, sure enough, it's much more efficient: https://imgur.com/a/epaKtDY. It seems like it's still early days in terms of how efficient self-play training is.
Related demos, repositories, papers and posts:
More recent and extensive post: https://blog.janestreet.com/deep-learning-the-hardest-go-pro...
Site to play against KataGo: https://online-go.com/player/592684/
Arxiv paper on KataGo design: https://arxiv.org/abs/1902.10565
GitHub repository of KataGo: https://github.com/lightvector/KataGo
Another writer commenting on the history of Go AI, and significance of KataGo: https://brantondemoss.com/writing/kata.html
He's also wrong about a lot. For all his insistence on accuracy, he himself is misleading or ignorant. Like his slam of the Dartmouth project - even if you knew nothing about it, all you have to do is click through to see the claims of 'solving vision' are sheer projection and urban legend. And he's happy to make up claims out of whole cloth: for example, when he says "AlphaGo works fine on a 19x19 board, but would need to be retrained to play on a rectangular board; the lack of transfer is telling.", for which he provides precisely zero evidence, he ignores the fact that AG training works fine on a mixture of board sizes and such progressive growing/curriculum training…
Read the full comment
in fact seems to accelerate training (https://arxiv.org/abs/1902.10565) and that rectangular convolutions are a thing that exist, and there would be plenty of transfer if anyone tried. If no one has tried that exact thing, it's because it would be pointless and it's obvious to everyone not named 'Marcus' that it'd work fine, being rectangular doesn't mystically stop it from working anymore than using a 13x13 rather than 19x19 board makes AG-style training stop working. (This is not the first time Marcus has claimed that something didn't work; I first realized that he doesn't actually keep up with the AI literature when I pointed out to him on Twitter that plenty of knowledge graphs were in use combined with DL, and Google Search was the biggest example of this, and he had no idea what I was talking about.)
No matter what DL or DRL does, or how little his own preferred paradigm does, Marcus will never ever admit anything. AlphaGo beats humans? Well, it just copied humans. AlphaZero learns from scratch? Well, he wrote a whole paper explaining how akshully it still copies humans because the tree search encodes the rules. MuZero throws out even the tree search's knowledge of rules? Crickets and essays about 'misinformation'.
AlphaGo Master, unsurprisingly, was significantly stronger than AlphaGoZero. AlphaZero, although it can play multiple games, was weaker yet. In both cases, they compared the 40 block version of the one with the 20 block version of the other (they had to double the network size to approach the level of the predecessor.)
Recently, Katago has reached similar levels of strength using a small fraction of the resources: https://arxiv.org/abs/1902.10565
It depends on what you mean by "more efficient." The significance of AlphaZero was that you can reach good results in a variety of domains even without human expert knowledge to provide supervised learning data or engineer features. It's efficient in terms of engineering resources.
A precisely tailored approach can always get better results.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.