Skip to content
Back to the collection

Mathematical Capabilities of ChatGPT

Simon Frieder, Luca Pinchetti, Alexis Chevalier

Show all 8 authors

Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Julius Berner

2023 · arXiv

Read the paper
HN score
0
Mentions
6
Readers
3
Months
4
What’s in a score?

From the abstract

We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

rvz
Neutral mention · rule
> The implications are vast. We may be able to translate between languages that have never had a “Rosetta Stone”. Any animals that have a true language could have it decoded. And while an LLM that’s gotten an 8 year old’s understanding of balancing assorted items isn’t that useful, an LLM that’s got a baby whale’s grasp on whale language would be revolutionary. Given that almost no-one trusts these LLMs for anything serious beyond creative / BS generating uses, this article does a lot of glossing over the reliability of their outputs and focuses more on such utopian promises with claims of 'understanding' which is in this article is beyond absurd and nonsensical. The…
Read the full comment
reality of LLMs is that they cannot reason. [0] It is known that LLMs perform incredibly worse in mathematical reasoning and understanding which it consistently hallucinates the wrong answers [1] and even when you tell it to correct itself, it pretends to identify its mistake and regurgitates another incorrect answer and its answers get even worse further down the conversation. [2] Of course to the typical user, it's easy to be fooled by a stochastic parrot pretending to demonstrate reasoning. But experts understand that these systems cannot transparently explain themselves and are AI sophists which fundamentally cannot be trusted. [0] https://arxiv.org/pdf/2302.08399.pdf [1] https://arxiv.org/pdf/2301.13867.pdf [2] https://arxiv.org/pdf/2310.01798.pdf
sfriedr
Neutral mention · rule
Could you share more about copyright? For example, aren't you worried that now, with all kinds of lawsuits happening [1] and copyright issues that were found in existing datasets [2], that you might get threatening letters from a lawyer some day? I'm the author of [3] where we introduced one of the first natural-language datasets that test graduate mathematics for LLMs, but some of the prompts we took from a copyrighted book and therefore thought about excluding them. Having them in the public dataset would be really nice though, hence I'm keen about your experience. I'd also be keen to hear how your challenge against the DMCA on sharing LLaMA's weights goes? [1] https://www.theguardian.com/books/2023/jul/05/authors-file-a... [2] https://arxiv.org/abs/2105.05241 [3] https://arxiv.org/abs/2301.13867
sfriedr
Neutral mention · rule
In February I published a paper on mathematics + ChatGPT (https://arxiv.org/abs/2301.13867) and my colleagues, who were also on LinkedIn, told me it garnered quite a bit of attention. So I signed up to LinkedIn on 3rd Feb. 2023, thinking that now would be a good time to step out of the academic ivory tower. The thing is that I used my Gmail account to sign into LinkedIn. Which worked fine and I was a happy user for about a month. I then went travelling and when I returned one month later to log in again with Gmail - suprise! A new account was created on the spot. So now I have the old account which had started to accumulate some important…
Read the full comment
contacts: https://ro.linkedin.com/in/simon-frieder-5b6874264?trk=publi... as well as a new, useless account: https://www.linkedin.com/in/simon-frieder-b7901b271/ I had been happy to use my Gmail account to login, so I did not bother to setup an extra email login process into LinkedIn. How can it be that the software engineering of a multi-million company even allows such blunders to happen?
sfriedr
Neutral mention · rule
Location: Oxford, UK Remote: Preferred Willing to relocate: Depending on location (inside EU preferred, but willing to consider other options depending on the work) Technologies: LLMs, other Deep Learning models (e.g. Vision Transformer), prompt engineering, PyTorch, LLMs, mathematical analysis Résumé/CV: On request (recent PhD in ML) Email: sfriedr_job3323@posteo.co Website: friederrr.org ........................ I offer machine/deep learning consultancy: happy to help you understand the state-of-art research and translate models implemented in recent papers into production code. I can only work part-time (either regularly a few days per week or larger but fixed blocks…
Read the full comment
of time), because I have ongoing research projects in machine learning that I need to finish, though I can work on these remotely if I need to relocate. I have two distinct graduate degrees, one in CS and one in pure mathematics and an eye for detail, which is crucial when it comes to doing a sensible implementation of certain neural networks. I also have expertise managing groups of data scientists and reseachers to pull of bigger scientific projects. My latest one was about about engineering and LLMs about which I'm particularly excited: https://arxiv.org/abs/2301.13867. It has been featured in the media, e.g. https://syncedreview.com/2023/02/03/genius-or-subpar-ai-math....
sfriedr
Neutral mention · rule
Location: Oxford, UK Remote: Preferred Willing to relocate: Depending on location (inside EU preferred, but willing to consider other options depending on the work) Technologies: LLMs, other Deep Learning models (e.g. Vision Transformer), prompt engineering, PyTorch, LLMs, mathematical analysis Résumé/CV: On request (recent PhD in ML) Email: frsi1@posteo.net Website: friederrr.org ........................ I offer machine/deep learning consultancy: happy to help you understand the state-of-art research and translate models implemented in recent papers into production code. I can only work part-time (either regularly a few days per week or larger but fixed blocks of time),…
Read the full comment
because I have ongoing research projects in machine learning that I need to finish, though I can work on these remotely if I need to relocate. I have two distinct graduate degrees, one in CS and one in pure mathematics and an eye for detail, which is crucial when it comes to doing a sensible implementation of certain neural networks. I also have expertise managing groups of data scientists and reseachers to pull of bigger scientific projects. My latest one was about about engineering and LLMs about which I'm particularly excited: https://arxiv.org/abs/2301.13867. It has been featured in the media, e.g. https://syncedreview.com/2023/02/03/genius-or-subpar-ai-math....

Keep following the thread.

More in Mathematics

More from the same topics.

1994arXiv

On proof and progress in mathematics

William P. Thurston

In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.

29 mentions21 months
9HN score
2017arXiv

The Case for Learned Index Structures

Tim Kraska, Alex Beutel, Ed H. Chi +2

Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not.

19 mentions14 months
5HN score