Solving Quantitative Reasoning Problems with Language Models

V3) breaks through and can actually solve a decent fraction of math problems, then reinforcement learning (plus other simple tricks like chain-of-thought prompting, simple ensemble voting, etc.) can easily juice the results through the following loop: 1) random search through different solution paths 2) identify the correct solution paths based on the final answer 3) train on the correct solution paths The exciting thing is that not only can RL bump up the performance of the current base model, but it can be used to generate new high-quality reasoning trace data, which was in painfully short-supply for training the initial models. This leads to a new wave of base models with better one-pass intuition, which leads to more efficient reinforcement learning search on harder problems, which leads to better training data... Note that this was basically impossible for non-LLM models in the past. You could always juice ImageNet classification performance with a simple ensemble of identically trained models, but that path didn't lead anywhere interesting because a juiced model didn't allow the creation of new synthetic data that was superior to the data it was trained on. The key difference is that LLMs not only output the solution but also output a solution path with all the intermediate steps - and these searched-and-filtered solution paths are much more valuable than the vast majority of the model's initial training data. [1] https://arxiv.org/abs/2305.20050 [2] https://arxiv.org/abs/2402.03300 and https://arxiv.org/abs/2206.14858

average middle-schooler. So even if you believe — rightly or wrongly — that current capabilities aren't very impressive, it's worth remembering that your understanding of current capabilities might not be entirely accurate, even if it's only a few months out of date. Incidentally, it may be worth looking at some examples of these models' outputs before deciding what they can or can't do. Here's Minerva solving some math problems, for example: https://dl.airtable.com/.attachmentThumbnails/32f10fc59039c7... https://dl.airtable.com/.attachmentThumbnails/f6f86e0edd14d1... https://dl.airtable.com/.attachmentThumbnails/d9bc743dca1015... https://dl.airtable.com/.attachmentThumbnails/51562ef70c2b14... https://dl.airtable.com/.attachmentThumbnails/c081906cc4d23b... I'll admit I find it challenging to interpret these results as "passable garbled nonesense [sic]", though perhaps I'm not being demanding enough. At any rate, when these models go from beating 10-year olds at math to beating 18-year olds at math in the span of 8 months, one does start to wonder how much of the hype is really due to over-interpretation — and what the next 8 months have in store. ======================== [1] https://arxiv.org/pdf/1709.04326.pdf (For comparison, a 3-time Mathematics Olympiad gold-medalist scored 90%.) [2] This was Poland's National Math Exam. Minvera scored 65%; the 2021 national average was 57%. See https://www.aitracker.org/ and Section 3 of the Minerva paper at https://arxiv.org/pdf/2206.14858.pdf [3] https://openai.com/blog/grade-school-math/

numbers is 10, and the variance of the first m even natural numbers is 16. Compute m + n". To preempt the "oh it is in the training set" issues, I'd note this (on a different problem): The model correctly takes the derivative of a position vector to compute the velocity. It then computes the norm of the resulting vector, and uses a trigonometric identity to obtain a final numerical answer. Minerva takes a completely different approach from the ground truth solution." I'd encourage you to read the Minerva paper[4] and update your understanding on what large neural networks are doing. These reasoning abilities are not* as simply as "compressed data representation + reconstruction" - the chain-of-reasoning approaches partially use that of course, but also appear to a form of lemma chaining, similar what something like Prolog does. As a specific demonstration for their ability to generate novel outputs that make sense: Prompt: explain Algorithm X in the style of a Kayne rap Response: Alright, here's a rap about Algorithm X[5], the famous algorithm for solving the exact cover problem: `Alright, here's a rap about Algorithm X, the famous algorithm for solving the exact cover problem: Verse 1: I'm Algorithm X, the greatest of them all I'll solve any exact cover problem, big or small My steps are simple, my logic is sound Just give me your matrix, and I'll give you the answer you found Chorus: Algorithm X, the master of the game I'll solve your problem, and do it by name Exact cover, no problem for me I'm Algorithm X, the MVP Verse 2: First, I'll start by selecting a column Then I'll look for a row with a one in that column Once I find a match, I'll mark the row and column And repeat the process until the matrix is gone Chorus: Algorithm X, the master of the game I'll solve your problem, and do it by name Exact cover, no problem for me I'm Algorithm X, the MVP Verse 3: But sometimes the matrix is too big to solve That's when I use my dancing moves I'll shuffle and prune, until the solution is clear And in the end, I'll have your answer, my dear Chorus: Algorithm X, the master of the game I'll solve your problem, and do it by name Exact cover, no problem for me I'm Algorithm X, the MVP.` Now I entirely concede that it has read things on how Algorithm X works, and that it seems to use a template for "rap" responses. But: But sometimes the matrix is too big to solve That's when I use my dancing moves I'll shuffle and prune, until the solution is clear And in the end, I'll have your answer, my dear I refuse to believe that anywhere, at any point has someone written an explanation of the use of dancing links[6] in Knuth's Algorithm X like that. [1] https://ai.googleblog.com/2022/11/characterizing-emergent-ph... [2] https://arxiv.org/abs/2204.02311 [3] https://ai.googleblog.com/2022/06/minerva-solving-quantitati... [4] https://arxiv.org/pdf/2206.14858.pdf [5] https://en.wikipedia.org/wiki/Knuth%27s_Algorithm_X [6] https://en.wikipedia.org/wiki/Dancing_Links