Skip to content
The methodOpen data
Back to the collection

Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath

Show all 8 authors

Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt

2021 · arXiv

MathematicsLanguage modelsMachine learning
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

tadamcz
Neutral mention · rule
Hi! I'm Tom, a machine learning engineer at the nonprofit research institute Epoch AI [0]. I've been working on building infrastructure to: * run LLM evaluations systematically and at scale * share the data with the public in a rigorous and transparent way We use the UK government's Inspect [1] library to run the evaluations. As soon as I saw this news on HN, I evaluated Mistral Small 3 on MATH [2] level 5 (hardest subset, 1,324 questions). I get an accuracy of 0.45 (± 0.011). We sample the LLM 8 times for each question, which lets us obtain less noisy estimates of mean accuracy, and measure the consistency of the LLM's answers. The 1,324*8=10,584 samples represent 8.5M…
Read the full comment
tokens (2M in, 6.5M out). You can see the full transcripts here in Inspect’s interactive interface: https://epoch.ai/inspect-viewer/484131e0/viewer?log_file=htt... Note that MATH is a different benchmark from the MathInstruct [3] mentioned in the OP. It's still early days for Epoch AI's benchmarking work. I'm developing a systematic database of evaluations run directly by us (so we can share the full details transparently), which we hope to release very soon. [0]: https://epoch.ai/ [1]: https://github.com/UKGovernmentBEIS/inspect_ai [2]: https://arxiv.org/abs/2103.03874 [3]: https://huggingface.co/datasets/TIGER-Lab/MathInstruct
Original comment Mistral Small 3
haltist
Neutral mention · rule
Part of an abstract from referenced articles: > Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.[1] 1: https://arxiv.org/abs/2103.03874
Original comment An AI Which Imitates Humans Can Beat Humans
nl
Neutral mention · rule
The most relevant reported benchmark results are on https://arxiv.org/abs/2103.03874 They compare it against PaLM and Minerva, both of which are fine tuned on math datasets. It outperforms both at roughly the same parameter count (eg Llama 65B vs Palm and Minerva 62B) but unclear how much of this is due to encoding vs the many other differences. It is useful to see that the performance increase is clearly not due to fine turning though.
Original comment Cerebras-GPT vs. LLaMA AI Model Performance Comparison

Keep following the thread.

More in Mathematics

More from the same topics.

1994arXiv

On proof and progress in mathematics

William P. Thurston

In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.

Mathematics
29 mentions21 months
9HN score