Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

I thought the AlphaZero paper was pretty cool: https://arxiv.org/abs/1712.01815 Not only did we get a whole new type of Chess engine, it was also interesting to see how the engine thought of different openings at various stages in its training. For instance, the Caro-Kann, which is my weapon of choice, was favored quite heavily by it for several hours and then seemingly rejected (perhaps it even refuted it?!) near the end.

According to the AlphaZero paper (https://arxiv.org/pdf/1712.01815.pdf) they ran game logic on TPUs: > Training proceeded for 700,000 steps (mini-batches of size 4,096) starting from randomly initialised parameters, using 5,000 first-generation TPUs to generate self-play games and 64 second-generation TPUs to train the neural networks. Further details of the training procedure are provided in the Methods.

>alphazero's loss function never changed Yes, it did. Alpha zero's loss function changes with every update. The loss function works on the actual outcome versus the predicted outcome, and the predicted outcome is a result of previous learning. So with every step it takes, which are random (and the initial weights are also random), it modifies it's own loss function for future walks in the space of weights. Rerun the entire training with different initial random weights, and look at the sequence of loss functions, and you will get a different sequence. The paper: https://arxiv.org/pdf/1712.01815.pdf

Not working in this area but the abstract of the AlphaZero paper [0] seems to disagree about your /any prior knowledge/ point: "Starting from random play, and given no domain knowledge except the game rules, AlphaZero achieved within 24 hours a superhuman level of play in the games of chess and shogi (Japanese chess) as well as Go, and convincingly defeated a world-champion program in each case." [0] https://arxiv.org/abs/1712.01815

The results were published - https://arxiv.org/abs/1712.01815 I agree AlphaZero had fancier hardware and so it wasn't really a fair fight.

There's a vector encoding "queen moves" and another encoding "knight moves". Between them, they cover all possible chess moves. Knight moves obviously are modelled by "knight moves". Edit: https://arxiv.org/pdf/1712.01815.pdf The information you want is on page 13, last paragraph. The previous paragraph describes the representation of the boardd.

I'm curious why you believe to be able to tell that my comment is not accurate when you yourself admit that you have no idea about deep learning? Note that I'm not saying that Google is doing something stupid or leaving potential gains on the table. What I'm saying is that their methods make sense when you are able to perform enough experiments to actually make data-driven decisions. There is just no way to emulate that when you don't even have the budget to try more than one value for some hyperparameters. And since you mentioned chess: The paper https://arxiv.org/pdf/1712.01815.pdf doesn't go into detail about hyperparameter tuning, but does say that they used Bayesian optimization. Although that's better than brute force, AFAIK its sample complexity is still exponential in the number of parameters.

Programming languages Slepak et al - An Array-Oriented Language with Static Rank Polymorphism [1] Distributed systems Kiayias et al - A Provably Secure Proof-of-Stake Blockchain Protocol [2] Cognitive computing Boahen - A Neuromorph's Prospectus [3] Machine learning Silver et al - Mastering Chess and Shogi [4] 1 http://www.ccs.neu.edu/home/shivers/papers/rank-polymorphism... 2 https://eprint.iacr.org/2016/889.pdf 3 https://pdfs.semanticscholar.org/3767/1e53c9949fa08d9dc150ad... 4 https://arxiv.org/pdf/1712.01815.pdf

Interesting information on reasons for choosing a 1GB hash size, thanks. > A0 should have let Stockfish use its opening book though. Has it been confirmed that it didn't? As far as I can tell, DeepMind never said that Stockfish didn't use an opening book or endgame tablebase, only that AlphaZero didn't: https://twitter.com/demishassabis/status/938347604462542849 The only time opening books are mentioned in the arxiv paper is when they are described as standard components of chess engines; while it doesn't explicitly state whether Stockfish was allowed to use them in the tests, I would think it likely: https://arxiv.org/abs/1712.01815 The rumor that Stockfish didn't have an opening book seems widespread but unsourced.