GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

mere words. English word strict has order, need it but actually not does. You skibidi rizz swag grok brah also, barely. We use language in the extended meaning of the word to create a model of the world and somehow the past riverrun skibidi transmits that model to others. This is what I believe Bender also tries to say in their paper. Now, you could claim that LLM’s have this deep structure, create models of the world and are basically just like us, and certainly many here are adament that this is the case, being aghast how someone can “insult” LLM’s by calling them parrots. However, there really is not much proof to back up this belief. Usually LLM’s seem to copy existing surface structure from whatever source, and when it deviates from these patterns, it usually becomes incomprehensible. There’s much hoopla about LLM’s solving hard maths, but it seems even there they are mostly generalising from vast amounts of training data, rather than actually reasoning: https://arxiv.org/pdf/2410.05229

they work internally, you will realize they do no reasoning whatsoever. Humans and many other animals are fully capable of reasoning outside of language (in the former case, prior to language acquisition), and the reduction of "intellgence" to "language" is a catagory error made by people falling vicim to the ELIZA effect[1], not the result of a sum of these particular statistical methods being equal real intelligence of any kind. 0: https://arxiv.org/pdf/2410.05229 1: https://en.wikipedia.org/wiki/ELIZA_effect

not actual intelligence matching the level of performance exhibited, i.e. no understanding. No person who can solve higher level math (like an LLM answering college or math olympiad questions) is confused by the kind of simple math blind spots that confuse LLMs. A person understanding higher level math, would never (and even less so, consistently) fail a problem like: "Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday, but five of them were a bit smaller than average. How many kiwis does Oliver have?" https://arxiv.org/pdf/2410.05229 (of course with these problems exposed, they'll probably "learn" to overfit it)

definition locked down before we begin to investigate the phenomenon or concept is precisely the wrong way to go about understanding it: we refine and rewrite the definitions as a consequence of what our investigations have discovered. Again, 'energy' provides a useful case study for how this happens. A third point about the word 'energy' is that it has become well-defined within physics, and yet retains much of its original vagueness in everyday usage, where, in addition, it is often used metaphorically. This is not a problem, except when someone makes the lexicographical fallacy of thinking that one can freely substitute the physics definition into everyday speech (or vice-versa) without changing the meaning. With many concepts about the mental, including 'reasoning', we are still in the learning-and-writing-the-definition stage. For example, let's take the definition you bring up: reasoning as good cognition. This just moves us on to the questions of what 'cognition' means, and what distinguishes good cognition from bad cognition (for example, is a valid logical argument predicated on what turns out to be a false assumption an example of reasoning-as-good-cognition?) We are not going to settle the matter by leafing through a dictionary, any more than Pedro Carolino could write a phrase book just from a Portugese-English dictionary (and you are probably aware that looking up definitions-of-definitions recursively in a dictionary often ends up in a loop.) A lot of people want to jump the gun on this, and say definitively either that LLMs have achieved reasoning (or general intelligence or a theory of mind or even consciousness, for that matter) or that they have not (or cannot.) What we should be doing, IMHO, is to put aside these questions until we have learned enough to say more precisely what these terms denote, by studying humans, other animals, and what I consider to be the surprising effectiveness of LLMs - and that is what the interviewee in the article we are nominally discussing here is doing. You entered this thread by saying (about the paper underlying an article in Ars Tech [1]) I’ll pop in with a friendly “that research is definitely wrong”. If they want to prove that LLMs can’t reason..., but I do not think there is anything like that claim in the paper itself (one should not simply trust what some person on HN says about a paper. That, of course, goes as much for what I say about it as what the original poster said.) To me, this looks like the sort of careful, specific and objective work that will lead to us a better understanding of our concepts of the mental. [1] https://arxiv.org/pdf/2410.05229