What is learned by sophisticated neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
> AlphaGo or AlphaZero didn’t need to model human cognition. It needed to see the current state and calculate the optimal path better than any human could.
I don't think this is right: To calculate the optimal path, you do need to model human cognition.
At least, in the sense that finding the best path requires figuring out human concepts like "is the king vulnerable", "material value", "rook activity", etc. We have actual evidence of AlphaZero calculating those things in a way that is at least somewhat like humans do:
https://arxiv.org/abs/2111.09259
So even chess has "hidden state" in a significant sense: you can't play well without calculating those values, which are far from the surface.
I'm not sure there's a clear line between chess and poker like the author assumes.
It looks like the basic meta-algorithm is:
1) Notice a pattern in the mathematics
2) Device a neural network that leverages structural properties of the space you wish to investigate
3) Encode relationship in a manner allowing use of supervised learning and see if the net can learn a pattern.
4) If it failed, rethink pattern and go back to step 2 or give up; else use attribution and explainability tools to try and extract human understandable concepts. Go back to step 2 until human converges
5) Use the extracted concepts to generate a conjecture or aid a proof.
6) Profit
In a way it's like AlphaZero in that a nnet is helping the searcher prune a highly branching decision…
Read the full comment
space.
Deepmind recently published another paper [for chess](https://arxiv.org/abs/2111.09259) which again was about extracting hidden knowledge from a neural network. Authors hope this can be an avenue through which machine learning can be useful to mathematics research. But even in the cases where their meta-algorithm works, I think they underestimate the difficulty of having expert neural network practitioners on hand, without which I expect step 3 failures will be massively inflated.
----------
Choice extracts:
> In each of these cases, the necessary models can be trained within several hours on a machine with a single graphics processing unit
> Further, in some domains the functions of interest may be difficult to learn in this paradigm. However, we believe there are many areas that could benefit from our methodology.
> The Bruhat interval of a pair of permutations is a partially ordered set of the elements of the group, and it can be represented as a directed acyclic graph. For modelling the Bruhat intervals, we use a message-passing neural network. We add two features at each node representing the in-degree and out-degree of that node.
> First, to gain confidence that the conjecture is correct, we trained a model to predict coefficients of the KL polynomial from the unlabelled Bruhat interval. We were able to do so across the different coefficients with reasonable accuracy giving some evidence that a general function may exist, as a four-step MPNN is a relatively simple function class. We trained a GraphNet model on the basis of a newly hypothesized representation and could achieve significantly better performance, lending evidence that it is a sufficient and helpful representation to understand the KL polynomial. To understand how the predictions were being made by the learned function f^, we used gradient-based attribution to define a salient subgraph SG for each example interval G
Why link to Chessbase instead of to the arxiv paper directly? https://arxiv.org/abs/2111.09259
Chessbase is notoriously sketchy and there is a lawsuit against them for blatantly stealing GPL code.
https://stockfishchess.org/blog/2021/our-lawsuit-against-che...
The paper itself is here: https://arxiv.org/abs/2111.09259 with the key conclusion that "Examining the evolution of human concepts using probing showed that many human concepts can be accurately regressed from the AZ network after training, even though AlphaZero has never seen a human game of chess, and there is no objective function promoting human-like play or activations" and "[t]he fact that human concepts can be located even in a superhuman system trained by self-play
broadens the range of systems in which we should expect to find human-understandable concepts"
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.