Anian Ruoss, Grégoire Delétang, Sourabh Medapati
Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, Tim Genewein
2024 · arXiv
This paper uses chess, a landmark planning problem in AI, to assess transformers' performance on a planning task where memorization is futile $\unicode{x2013}$ even at a large scale. To this end, we release ChessBench, a large-scale benchmark dataset of 10 million chess games with legal move and value annotations (15 billion data points) provided by Stockfish 16, the state-of-the-art chess engine. We train transformers with up to 270 million parameters on ChessBench via supervised learning and perform extensive ablations to assess the impact of dataset size, model size, architecture type, and different prediction targets (state-values, action-values, and behavioral cloning). Our largest models learn to predict action-values for novel boards quite accurately, implying highly non-trivial generalization. Despite performing no explicit search, our resulting chess policy solves challenging chess puzzles and achieves a surprisingly strong Lichess blitz Elo of 2895 against humans (grandmaster level). We also compare to Leela Chess Zero and AlphaZero (trained without supervision via self-play) with and without search. We show that, although a remarkably good approximation of Stockfish's search-based algorithm can be distilled into large-scale transformers via supervised learning, perfect distillation is still beyond reach, thus making ChessBench well-suited for future research.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
19 selected excerpts · 19 total mentions
I meant trivial in the sense it's a solved problem, I'm sure it still costs a non-negligible amount of money to train it. See for example the chess transformer built by DeepMind a couple of years ago which I referred to in a sibling comment [1]. [1] https://arxiv.org/abs/2402.04494
>These models cannot even make legal chess moves. That’s incredibly basic logic, and it shows how LLMs are still completely incapable of reasoning or understanding. Yeah they can. There's a link I shared to prove it which you've conveniently ignored. LLMs learn by predicting, failing and getting a little better, rinse and repeat. Pre-training is not like reading a book. LLMs trained on chess games play chess just fine. They don't make the silly mistakes you're talking about and they very rarely make illegal moves. There's gpt-3.5-turbo-instruct which i already shared and plays at around 1800 ELO. Then there's this grandmaster level chess transformer -…
https://arxiv.org/abs/2402.04494. They're also a couple of models that were trained in the Eleuther AI discord that reached about 1100-1300 Elo. I don't know what the peak of LLM Chess playing looks like but this is clearly less of a 'LLMs can't do this' problem and more 'Open AI/Anthropic/Google etc don't care if their models can play Chess or not' problem. So are they capable of reasoning now or would you like to shift the posts ?
I am an MLE not an expert. However, it is a fundamental problem that our current paradigm of training larger and larger LLMs cannot ever scale to the precision people require for many tasks. Even in the highly constrained realm of chess, an enormous neural net will be outclassed by a small program that can run on your phone. https://arxiv.org/pdf/2402.04494
There are several neural network based engines these days, including one that does exclusively what you describe (i.e. "patterns only", no calculation at all), and one that's trained on human games. Even Stockfish uses a neural network these days by default for its positional evaluation, but it's relatively simple/lightweight in comparison to these, and it gains its strength from being used as part of deep search, rather than using a powerful/heavy neural network in a shallow tree search. [1] https://arxiv.org/html/2402.04494v1 [2] https://www.maiachess.com/
I suppose you didn't get the news, but google developed a LLM that can play chess. And play it at grandmaster level: https://arxiv.org/html/2402.04494v1
The news that got my attention was https://arxiv.org/html/2402.04494v1 "Grandmaster-Level Chess Without Search" and an engine using this approach exceeding 2900 on Lichess (against humans) and over 2200 against engines. I actually started a chess channel a couple of hourse ago to help humans take advantage of this (nothing there yet https://youtube.com/@DecisiveEdgeChess ). I have long taught my students that its possible to assess positions at a very high level with very, very little calculation, and this news hit me as "finally, evidence enough to intrigue more people that this is possible!" (My interest in Chess goes way back. I finished in the money in the U.S. Open…
and New York Open in the '80's, and one of my longtime friends was IM Mike Valvo, since passed, who was the arbiter for the 1996 match between Garry Kasparov and IBM's Deep Blue, and a commentator for the '97 match alongside Grandmasters Yasser Seirawan and Maurice Ashley.)
A recent discovery showing that computer chess (which was traditionally based on searching a combinatorial space, i.e. NP-hard) that is instead now being solved with transformer models, is actually playing better at the ELO level. https://arxiv.org/pdf/2402.04494 If you think about using search to play chess, it can go several ways. Brute-forcing all chess moves (NP hard) doesn’t work because you need almost infinite compute power. If you use a chess engine with clever heuristics to eliminate bad solutions, you can solve it in finite time. But if you learn from the best humans performing under different contexts (transformers are really good at capturing context in sequences and predicting the next token from that context — hence their utility in LLMs) you have narrowed your search space even further to only set of good moves (by grandmaster standards).
So I guess when LLMs play chess at 2800 ELO (https://arxiv.org/pdf/2402.04494) they are just taking a vote on what top grand masters would play in the given position. I wonder how you would explain it when they play above any human level.
https://arxiv.org/abs/2402.04494 > Board states s are encoded as FEN strings which we convert to fixed-length strings of 77 characters where the ASCII-code of each character is one token. A FEN string is a description of all pieces on the board, whose turn it is, the castling availability for both players, a potential en passant target, a half-move clock and a full-move counter. We essentially take any variable-length field in the FEN string, and convert it into a fixed-length sub-string by padding with ‘.’ if needed. We never flip the board; the FEN string always starts at rank 1, even when it is the black’s turn. We store the actions in UCI notation (e.g., ‘e2e4’ for…
the well-known white opening move). To tokenize them we determine all possible legal actions across games, which is 1968, sort them alphanumerically (case-sensitive), and take the action’s index as the token, meaning actions are always described by a single token (all details in Section A.1). I am starting to notice a pattern in these papers - Writing hyper-specific tokenizers for the target problem. How would this model perform if we made a small change to the rules of chess and continued using the same tokenizer? If we find we need to rewrite the tokenizer for every problem variant, then I argue this is just ordinary programming in a very expensive disguise.
Experimenting applying Meta's V-JEPA [0] architecture for representation learning to chess. One of the challenges is that validating if the model is learning useful dynamics of the game, so I'm using it as an excuse to learn some reinforcement learning by using the representations generated by the JEPA model to approximate useful Q-values [1]. This method currently has no search so I'm planning on comparing with this paper [2] which achieves GM level chess without any search. Honestly, Im unsure if the full pipeline is stable enough to even converge, but it's fun experimenting. I'm bad at chess so I really want to make a bot that challenges the best bots on lichess. [0]: https://ai.meta.com/research/publications/revisiting-feature... [1]: https://en.wikipedia.org/wiki/Q-learning [2]: https://arxiv.org/abs/2402.04494
What transformers have taught us is that gobbling up the whole internet can get you pretty far, but it doesn't change the fundamentals. Hallucinations still are trivial [1] to elicit. On the other hand, transformers are a software breakthrough that made training on the whole internet even feasible. Other methods (RNNs and LSTMs) were simply too slow. Transformers are a new hammer for an internet-sized nail. If I can make an analogy to chess engines, a type of AI so old that people don't call it AI anymore [2], if we had started out by applying transformers to available chess games, we would never be able to create an engine that could reliably beat a grandmaster.…
Predicting the next chess move based on some representation of the game and its relationship with previously-seen positions only gets you so far. Deep Mind tried [3] this. One route forward, which doesn't require a fundamental breakthrough, would be synthetic data. To stick with chess, we could create huge synthetic datasets of a huge amount of chess games, and simply continue training as we were. However, this is very expensive, and can be tricky if the synthetic distribution isn't the same as the real one. And the end result is still going to require incredible computation power to use. What an LLM really lacks is the ability to search. And search is really important [4]: without it you'll probably need orders of magnitude more data and orders of magnitude larger models. At a fundamental level, search means we don't just make a snap judgement about e.g. the likely next token, but, as Daniel Kahneman would say, "thinking slowly" about what to do/say next. Assuming we will not have search is how you get projections like needing 100 GW power plants to supply dedicated training data centers. The human brain uses 12 watts. 100 GW is the energy requirements of all human brains on the planet. And yet, if we put all human brains on the planet together, our collective capabilities would be outclassed by a chess engine I can run on my phone. [1] https://arxiv.org/pdf/2406.02061 [2] https://en.wikipedia.org/wiki/AI_effect [3] https://arxiv.org/pdf/2402.04494 [4] https://www.youtube.com/watch?v=ceCg90Q9N6Y&t=1200s
People have advanced that argument a lot, and it's often worked for a short while; then the statistical models get better. Chess was a game for humans. It was very briefly a game for humans and machines (Kasparov had a go at getting "Advanced Chess" off the ground as a competitive sport), but soon enough having a human in the team made the program worse. But at least the evaluation functions were designed by humans, right? That lasted a remarkably long time; first Stockfish became the strongest engine in the world by using distributed hyperparameter search to tweak its piece-square tables, then AlphaZero came along and used a policy network + MCTS instead of alpha-beta…
search, then (with an assist from the Shogi community) Stockfish struck back with a completely learned evaluation function via NNUE. So the last frontier of human expertise in chess is search heuristics, and that's going to fall too: https://arxiv.org/abs/2402.04494. The common theme with all of this is that the stuff which we used before are, fundamentally, hacks to get around _not having enough compute_, but which make the system worse once you don't have to make those tradeoffs around inductive biases. Empirical evidence suggests that raw scaling has a long way to run yet.
The big problem with that paper is that grandmaster level chess is actually really bad for a chess bot. Its elo is a full 400 lower than Stockfish[0], which is the same distance between a grandmaster and a very strong untitled player. Oddly enough, I wrote a blog post a few months ago that's tangentially related to this topic[1]. The gist of it is that the search tree for a chess engine is the most important part, since a perfect solution for chess would just be a brute-force search of the entire game tree. Trying to get around that by building a better evaluation function is just a waste of time and money. [0] https://arxiv.org/pdf/2402.04494.pdf [1] https://matteoraso.github.io/board-game-engines-are-about-tr...
There are chess engines based on transformers, even DeepMind released one [1]. It achieved ~2900 Elo. It does have peculiarities for example in the endgame that are likely derived from its architecture, though I think it definitely qualifies as an example of the fact that simply because something is a next token predictor doesn't mean it cannot perform tasks that require intelligence and planning. The r in strawberry is more of a fundamental limitation of our tokenization procedures, not the transformer architecture. We could easily train a LLM with byte-size tokens that would nail those problems. It can also be easily fixed with harnessing (ie for this class of problems, write a script rather than solve it yourself). I mean, we do this all the time ourselves, even mathematicians and physicists will run to a calculator for all kinds of problems they could in principle solve in their heads. [1] https://arxiv.org/abs/2402.04494
That might be overstating it, at least if you mean it to be some unreplicable feat. Small models have been trained that play around 1200 to 1300 on the eleuther discord. And there's this grandmaster level transformer - https://arxiv.org/html/2402.04494v1 Open AI, Anthropic and the like simply don't care much about their LLMs playing chess. That or post training is messing things up.
In Google's paper, then titled "Grandmaster level chess without search", they evaluate turbo-instruct to have a lichess Elo of 1755 (vs bots) https://arxiv.org/abs/2402.04494 Admittedly, this isn't really "the source" though. The first people to break the news on turbo-instruct's chess ability all pegged it around 1800. https://x.com/GrantSlatton/status/1703913578036904431
This repository provides an implementation of our paper Grandmaster-Level Chess Without Search. https://arxiv.org/abs/2402.04494 The recent breakthrough successes in machine learning are mainly attributed to scale: namely large-scale attention-based architectures and datasets of unprecedented scale. This paper investigates the impact of training at scale for chess. Unlike traditional chess engines that rely on complex heuristics, explicit search, or a combination of both, we train a 270M parameter transformer model with supervised learning on a dataset of 10 million chess games. We annotate each board in the dataset with action-values provided by the powerful Stockfish 16…
engine, leading to roughly 15 billion data points. Our largest model reaches a Lichess blitz Elo of 2895 against humans, and successfully solves a series of challenging chess puzzles, without any domain-specific tweaks or explicit search algorithms. We also show that our model outperforms AlphaZero's policy and value networks (without MCTS) and GPT-3.5-turbo-instruct. A systematic investigation of model and dataset size shows that strong chess performance only arises at sufficient scale. To validate our results, we perform an extensive series of ablations of design choices and hyperparameters.
>An LLM will learn what it CAN (and needs to to reduce the loss), but not what it CAN'T. How difficult is that to understand?! Right and the point is that you don't know what it CAN'T learn. You clearly don't quite understand this because you say stuff like this: >Chess is a good example, since it's easy to understand. The generative process for world class chess (whether human, or for an engine) involves way more DEPTH (cf layers) of computation than the transformer has available to model it and it's just baffling because: 1. Humans don't play chess anything like chess engines. They literally can't because the brain has iterative computation limits well below that of a…
computer. Most Grandmasters are only evaluating 5 to 6 moves deep on average. 2. We have a chess transformer playing world class chess (grandmaster level) - https://arxiv.org/abs/2402.04494. You keep trying to make the point that because a Transformer architecturally has a depth limit for some trained model, a, it cannot reach human level. But this is nonsensical for various reasons. - Nobody is stopping you from just increasing N such that every GI problem we care about is covered. - You have shown literally no evidence that the N even state of the art models posses today is insufficient to match human iterative compute ability. GPT-4o instant shots arbitrary arithmetic more accurately than any human brain and that's supposedly something it's bad at. You can clearly see it can reach world class chess play. If you have some iterative computation benchmark that shows transformers zero shotting worse than an unaided human then feel free to show me.
I’m curious about the comparison between NNUE and the LLM-based model that Deep Mind announced a couple of weeks ago (https://arxiv.org/pdf/2402.04494.pdf). Using NNUE only (i.e. depth 1 search) would be directly comparable. If Deep Mind’s model is better it raises interesting questions about scaling laws for this kind of thing.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.