Skip to content
The methodOpen data
Back to the collection

Complete Chess Games Enable LLM Become A Chess Master

Yinqi Zhang, Xintian Han, Haolong Li, Kedi Chen, Shaohui Lin

2025 · arXiv

Language models
Read the paper
HN score
0
Mentions
4
Readers
3
Months
3
What’s in a score?

From the abstract

Large language models (LLM) have shown remarkable abilities in text generation, question answering, language translation, reasoning and many other tasks. It continues to advance rapidly and is becoming increasingly influential in various fields, from technology and business to education and entertainment. Despite LLM's success in multiple areas, its ability to play abstract games, such as chess, is underexplored. Chess-playing requires the language models to output legal and reasonable moves from textual inputs. Here, we propose the Large language model ChessLLM to play full chess games. We transform the game into a textual format with the best move represented in the Forsyth-Edwards Notation. We show that by simply supervised fine-tuning, our model has achieved a professional-level Elo rating of 1788 in matches against the standard Elo-rated Stockfish when permitted to sample 10 times. We further show that data quality is important. Long-round data supervision enjoys a 350 Elo rating improvement over short-round data.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

famouswaffles
Neutral mention · rule
Relax. Anyone who's genuinely interested in the question will see with a few searches that LLMs can play chess fine, although the post-trained models mostly seem to be regressed. Problem is people are more interested in validating their own assumptions than anything else. https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2501.17186 https://github.com/adamkarvonen/chess_gpt_eval
Original comment Stop Thinking of AI as a Coworker. It's an Exoskeleton
tmnvdb
Neutral mention · rule
If you train an LLM on chess, it will learn that too. You don't need to explain the rules, just feed it chess games, and it will stop making illegal moves at some point. It is a clear example of an inferred world model from training. https://arxiv.org/abs/2501.17186 PS "Major commercial American LLM" is not very meaningful, you could be using GPT4o with that description.
Original comment LLMs aren't world models
famouswaffles
Neutral mention · rule
>They still make these errors on anything that is out of distribution. There is literally a post in this thread linking to a chat where Sonnet failed a basic arithmetic puzzle: https://news.ycombinator.com/item?id=47051286 I thought we were talking about actual arithmetic not silly puzzles, and there are many human adults that would fail this, nevermind children. >LLMs can match an example at exactly that trivial level because it can be predicted from context. However, if you construct a more complex example with several rules, especially with rules that have contradictions and have specified logic to resolve conflicts, they fail badly. Even if that were true (Have you…
Read the full comment
actually tried?), You do realize many humans would also fail once you did all that right ? >They can't even reliably play Chess or Poker without breaking the rules despite those extremely well-represented in the dataset already, nevermind a made-up set of logical rules. LLMs can play chess just fine (99.8 % legal move rate, ~1800 Elo) https://arxiv.org/abs/2403.15498 https://arxiv.org/abs/2501.17186 https://github.com/adamkarvonen/chess_gpt_eval
Original comment Claude Sonnet 4.6 System Card

Keep following the thread.

More in Language models

More from the same topics.