Skip to content
Back to the collection

ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems

Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, Henry Pinkard

2025 · arXiv

Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), introduced in 2019, established a challenging benchmark for evaluating the general fluid intelligence of artificial systems via a set of unique, novel tasks only requiring minimal prior knowledge. While ARC-AGI has spurred significant research activity over the past five years, recent AI progress calls for benchmarks capable of finer-grained evaluation at higher levels of cognitive complexity. We introduce ARC-AGI-2, an upgraded version of the benchmark. ARC-AGI-2 preserves the input-output pair task format of its predecessor, ensuring continuity for researchers. It incorporates a newly curated and expanded set of tasks specifically designed to provide a more granular signal to assess abstract reasoning and problem-solving abilities at higher levels of fluid intelligence. To contextualize the difficulty and characteristics of ARC-AGI-2, we present extensive results from human testing, providing a robust baseline that highlights the benchmark's accessibility to human intelligence, yet difficulty for current AI systems. ARC-AGI-2 aims to serve as a next-generation tool for rigorously measuring progress towards more general and human-like AI capabilities.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

therobots927
Neutral mention · rule
I was meaning to imply that yes assuming we had a proportionate amount of birdsong data, would we be able to reverse engineer their flight abilities. I think given the fact that spatial reasoning is nearly universal among species, we can very safely assume that it is “too evolutionary old and too low-level, too far removed from higher cognition, to leak into language heavily” I think this is pretty apparent. It’s very rare for athletes to talk through their actions in high level detail - I saw the ball coming towards me at a 37 degree phi 23 degree epsilon angle at a speed of approximately 20 mph, I estimated it’s time to arrival would be .45 seconds etc. The eye-hand…
Read the full comment
coordination occurs almost completely outside of what you consider conscious awareness. And it’s not easy to describe that’s why athletic coaching is difficult to do through words alone. As far as ARC-AGI goes it looks like last years models were scoring <5% against their v2 benchmark: https://arxiv.org/pdf/2505.11831 Frankly I don’t understand why you can’t train a multi-modal LLM on video game frame data. Is that just way too compute intensive to do? What am I missing here? Because I think it’s crazy to think that an LLM could learn to think spatially just from reading… even if they’re reading everything that’s ever been written. I think that about summarizes my position.
westurner
Neutral mention · rule
"ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems" (2025) https://arxiv.org/abs/2505.11831 ARC Prize: $1m: https://arcprize.org/ : > At ARC Prize, our mission is to serve as a North Star towards AGI through enduring benchmarks, directing efforts towards systems capable of general intelligence and significantly compressing the timeline for scientific breakthroughs ARC-AGI-3: Interactive Reasoning Benchmark (2026) https://arcprize.org/arc-agi/3/ : > The first eval that measures human-like intelligence in AI. [w/ HuggingFace]
yorwba
Neutral mention · rule
The ARC-AGI-2 paper https://arxiv.org/pdf/2505.11831#figure.4 uses a non-representative sample, success rate differs widely across participants and "final ARC-AGI-2 test pairs were solved, on average, by 75% of people who attempted them. The average test-taker solved 66% of tasks they attempted. 100% of ARC-AGI-2 tasks were solved by at least two people (many were solved by more) in two attempts or less." Certainly those non-representative humans are much better than current models, but they're also far from scoring 100%.

Keep following the thread.

More in Psychology

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

16 mentions14 months
2.5HN score