Skip to content
The methodOpen data
Back to the collection

Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks

Melanie Mitchell, Alessandro B. Palmarini, Arseny Moskvichev

2023 · arXiv

Language models
Read the paper
HN score
0
Mentions
6
Readers
3
Months
4
What’s in a score?

From the abstract

We explore the abstract reasoning abilities of text-only and multimodal versions of GPT-4, using the ConceptARC benchmark [10], which is designed to evaluate robust understanding and reasoning with core-knowledge concepts. We extend the work of Moskvichev et al. [10] by evaluating GPT-4 on more detailed, one-shot prompting (rather than simple, zero-shot prompts) with text versions of ConceptARC tasks, and by evaluating GPT-4V, the multimodal version of GPT-4, on zero- and one-shot prompts using image versions of the simplest tasks. Our experimental results support the conclusion that neither version of GPT-4 has developed robust abstraction abilities at humanlike levels.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

lossolo
Neutral mention · rule
Your entire edit essentially walks back your earlier strong claims. None of your current points actually support your position. 1. No, it doesn't. That's a ridiculous claim. Are you seriously suggesting that statistics require reasoning? 2. If you map that language to tokens, it's obvious the model will follow that mapping. etc. Here are papers showing that these models can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809 You're mistaking pattern matching and the modeling of relationships in latent space for genuine reasoning. I don't know what…
Read the full comment
you're working on, but while I'm not curing cancer, I am solving problems that aren't in the training data and can't be found on Google. Just a few days ago, Gemini 2.5 Pro literally told me it didn’t know what to do and asked me for help. The other models hallucinated incorrect answers. I solved the problem in 15 minutes. If you're working on yet another CRUD app, and you've never implemented transformers yourself or understood how they work internally, then I understand why LLMs might seem like magic to you.
Original comment Seven replies to the viral Apple reasoning paper and why they fall short
skepticATX
Neutral mention · rule
There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your programming language). One benchmark that I track closely is ConceptARC, which aims to test generalization and abstraction capabilities. Here is a very recent result that uses the benchmark: https://arxiv.org/abs/2311.09247. Humans correctly solved 91% of the problems, GPT-4 solved 33%, and GPT-4V did much worse than GPT-4.
Original comment I disagree with Geoff Hinton regarding "glorified autocomplete"
lossolo
Neutral mention · rule
> You can get them to solve unseen problems just fine Prove that the problem wasn't seen by them in other form. > Specify a grammar in BNF notation and tell it to generate or parse sentences for you. You can produce a more than random enough grammar that it it can't have derived the parsing of it from past text, but necessarily reasons about BNF notation sufficiently well to be able to use it to deduce the grammar, and use that to parse subsequent sentences. You can have it analyse them and tag them according to the grammar to. And generate sentences. Oh, come on. It's like rewriting the same program in another programming language with different variables. What it can't…
Read the full comment
do is to create a concept of programming language, I'm not talking about a new programming language, I'm talking about the concepts. > I'm sure that won't work for every problem of this sort, but the quality of tests people do on LLMs is really awful, at least because people tend to do very narrow tests like that and make broad pronouncements about what LLM's "can't" do based on it. Here, a few papers that show they can't reason: https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809
Original comment Markov chains are funnier than LLMs
lossolo
Neutral mention · rule
> "generalize to its dataset" is a contradiction Not "to" but over, example the same code written in one language over the other language. > if you think being able to generalize in ways similar to the whole of the internet does not give your meaningful abilities to reason, I'm not sure what I can tell you If after reading papers below that show empirically that they can't reason, you will still think they can reason, then I don't know what I can tell you. https://arxiv.org/abs/2311.00871 https://arxiv.org/abs/2309.13638 https://arxiv.org/abs/2311.09247 https://arxiv.org/abs/2305.18654 https://arxiv.org/abs/2309.01809
Original comment Markov chains are funnier than LLMs
lossolo
Neutral mention · rule
A few more interesting papers not mentioned in the article: "Faith and Fate: Limits of Transformers on Compositionality" https://arxiv.org/abs/2305.18654 "Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Tasks": https://arxiv.org/abs/2311.09247 "Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve" https://arxiv.org/abs/2309.13638 "Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models" https://arxiv.org/abs/2311.00871
Original comment Will scaling work?

Keep following the thread.

More in Language models

More from the same topics.