Zhaofeng Wu, Linlu Qiu, Alexis Ross
Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, Yoon Kim
2023 · arXiv
The impressive performance of recent language models across a wide range of tasks suggests that they possess a degree of abstract reasoning skills. Are these skills general and transferable, or specialized to specific tasks seen during pretraining? To disentangle these effects, we propose an evaluation framework based on "counterfactual" task variants that deviate from the default assumptions underlying standard tasks. Across a suite of 11 tasks, we observe nontrivial performance on the counterfactual variants, but nevertheless find that performance substantially and consistently degrades compared to the default conditions. This suggests that while current LMs may possess abstract task-solving skills to an extent, they often also rely on narrow, non-transferable procedures for task-solving. These results motivate a more careful interpretation of language model performance that teases apart these aspects of behavior.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
7 selected excerpts · 7 total mentions
Is there something more than prediction going on? I'd like to understand why there would be studies like this: https://arxiv.org/pdf/2307.02477 if it's all just prediction. Are there some algorithms that are called "reasoning", that are somewhere between prediction and full-blown reasoning?
>Your last linked paper has flawed methodology for what it attempts to demonstrate, as shown in this paper: https://arxiv.org/pdf/2307.02477 Genuinely, What's wrong with the methodology? Your paper literally admits humans would also perform worse at counterfactuals. Worse than a LLM ? Maybe not but it never bothers to test this so... The problem here is that none of the definitions (those that are testable) so far given actually separate humans from LLMs. They're all tests some humans would also flounder at or that LLMs perform far greater than chance at, if below some human's level. If you're going to say, "LLMs don't do real reasoning because of x" then x better be something all humans clear if what humans do is "real reasoning". Humans perform worse at counterfactuals so saying "Hey, see this paper that shows LLMs doing the same, It means they don't reason" is a logical fallacy if you don't extend that conclusion to humans as well.
It appears that you are the only person in this discussion making many incorrect assumptions. Based on your comments, I would assume you are actually googling those papers based on their abstracts. Your last linked paper has flawed methodology for what it attempts to demonstrate, as shown in this paper: https://arxiv.org/pdf/2307.02477 The tests you're requesting are provided within the previously linked papers. I'm not sure what you want. Do you expect people to copy and paste entire papers here that show methodology and describe experiments? You wrote, "I'm asking you to define 'real reasoning'," which is actually defined in the blog post linked earlier in this discussion. In fact, the entire blog post is about this topic. It appears that you are not thoroughly reading the material. Your replies resemble those of a human stochastic parrot.
Same, I've been pretty impressed as well and typically give Claude a shot. Sometimes I even pass their results back and forth in an LLM collab so they generate more diverse perspectives. However, this paper from 4 days ago shows that Claude can fall apart quickly in out of distribution tasks. If you ask opposite day questions, GPT-4 is weirdly strong at it (figure 2). https://arxiv.org/pdf/2307.02477
Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'. As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Literally, everything here is arbitrary convention; and under arbitrary permutation, the sign can mean the same, and share no properties with any other permutation whatsoever. Rather it is by being-in-the-world you have acquired a direct understanding of roads, speeds, limits, rules, laws, society,…
etc. And alongside this all the while internalizing arbitary conventions by which we signify these things. A stop sign could look like anything at all. We just decide to use arabic numerals, red, white, a certain eye-hight, etc. The meaning of the sign is something you are reconstructing prior even to being cognitively aware of the sign you're looking at. The meaning of these conventions is a mental reconstruction made possible by your direct familiarity with the world these conventions are related to. You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs. There is literally nothing at all in speed signs that has anything to do with their meaning. This isnt a speculative observation. It places a very hard limit on what the entire field of statistical AI can do, let alone LLMs. Any model created by empirical risk minimization, or fitting-to-historical-data, lacks any relevant capacities. It has only one: drawing a data point from some inferred distribution over historical cases. In any case, this isnt a speculative exercise. Statistical AI will be forever "limited" to a fixed computational budget for computing predictions over historical cases, with necessarily limitations this involves -- this is only an engineering problem where these cases aren't representative of the kinds of queries people have. eg., they will fail under relevant counter-factual permutation of the conventions of the language/domain/topic, etc. -- see, e.g., https://arxiv.org/pdf/2307.02477 Applying the lessons of the above paper, note chatgpt3.5 gets the evaluation of python wrong under permuation of its syntax: https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702... ..whereas ChatGptv4o does not. This suggests to me that there's been a very large amount of prompting by users since 3.5 trying to create novel programming languges in this way.. hence 4o, having been trained on subsequent user prompts, its "better" at this task. Of course nothing important has changed between v3.5 and v4o in terms of what the model is, rather there is more query-relevant data to sample from.
I think it regurgitates badly, an interesting pre-print is the "reasoning or reciting" one from https://arxiv.org/abs/2307.02477 But - it's only a pre-print and I don't think that they have taken it forward to a publication so handle with care. Anyway, the relevant evidence and thinking that I would highlight and somewhat agrees with my findings is seen in figure 7 and section 5.6. I struggle to get the quality of results that they saw, but its believed by some people that ongoing development of GPT-4 and throttling of the reasoning for cost reasons may have limited some of its capabilities by the end of this year so that may be some of my problem.
Exactly. Current LLMs fall over when facing counterfactuals: https://arxiv.org/abs/2307.02477. This is why it's mostly meaningless to for a LLM to pass the bar, but not meaningless for a human to do so. We (rightly, for the most part) assume that a human who passes the bar can transfer those skills into unique and novel situations. We can't make that assumption for LLMs, because they are lacking adaptability that is needed for true intelligence.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.