Skip to content
The methodOpen data
Back to the collection

LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench

Karthik Valmeekam, Kaya Stechly, Subbarao Kambhampati

2024 · arXiv

PhysicsLanguage models
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities. PlanBench, an extensible benchmark we developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating the planning abilities of LLMs. Despite the slew of new private and open source LLMs since GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs--making it a new kind of model: a Large Reasoning Model (LRM). Using this development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new LRMs do on PlanBench. As we shall see, while o1's performance is a quantum improvement on the benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings to the fore questions about accuracy, efficiency, and guarantees which must be considered before deploying such systems.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

YeGoblynQueenne
Neutral mention · rule
>> Large language models (LLMs) have demonstrated incredible reasoning abilities, penetrating an increasing number of domains in our lives. This is now established orthodoxy, a bit like astrology in ancient times but it is complete nonsense. Nope, LLMs have not demonstrated any credible, or incredible reasoning abilities. They have demonstrated an excellent ability for approximate retrieval of previously observed answers (with variations, which should not surprise anyone given that those are generative models) but they fail spectacularly when they have to "reason" in contexts where they really can't have seen the answer anywhere before. For example, the "randomised…
Read the full comment
mystery blocksworld" from this paoer: LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench https://arxiv.org/abs/2409.13373 "Randomised Mystery Blocksworld" is a version of the good old blocksworld planning benchmark where the names of obejcts and actions have been changed to random strings. The vast majority of LLMs score pathetically low in this, but much better in non-randomised versions, very clearly demonstrating excellent memorisation skills, but pitiful reasoning ability. As you 'd expect from a, well, language. model. >> A possible explanation for this is that the thought tokens allow the model to explore multiple possible branches before committing to a specific path, whereas chain-of-thought reasoning chooses a direction from the start. This ability is somewhat similar to Breadth-First Search (BFS). Why BFS in particular? Why not DFS or A*? I can't see any breadth-first bias in those graphs. BFS is not the only graph-traversing algorithm.
Original comment Coconut by Meta AI – Better LLM Reasoning with Chain of Continuous Thought?
roboboffin
Neutral mention · rule
For example, papers like this call into question whether or not a LLM can plan: https://arxiv.org/html/2409.13373v1 This is a basic form of reasoning, to plan out the steps needed to execute something.
Original comment Were RNNs all we needed?
mrbungie
Neutral mention · rule
Last time I checked, o1 did much better in PlanBench that models from other providers indicating better perf in that area [1][2]. Probably that lead will rapidly decay as other players implement similar approaches to "reasoning"/LRMs. [1] https://www.arxiv.org/abs/2409.13373 [2] https://x.com/omarsar0/status/1838353480672563581
Original comment Apple No Longer in Talks to Join OpenAI Investment Round

Keep following the thread.

More in Physics

More from the same topics.