Skip to content
The methodOpen data
Back to the collection

Large Language Models are Zero-Shot Reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa

2022 · arXiv

Language modelsNeuroscience
Read the paper
HN score
0
Mentions
13
Readers
12
Months
13
What’s in a score?

From the abstract

Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding "Let's think step by step" before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with large InstructGPT model (text-davinci-002), as well as similar magnitudes of improvements with another off-the-shelf large model, 540B parameter PaLM. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted by simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

13 selected excerpts · 13 total mentions

CamperBob2
Neutral mention · rule
These arguments usually seem to come down to disagreements about definitions, as you suggest. You've talked about what you don't consider evidence of intelligence, but you haven't said anything about the criteria you would apply. What evidence of intelligent reasoning would change your mind? It is unsupportable to claim that ML researchers at leading labs share your opinion. Since roughly 2022, they understand that they are working with systems capable of reasoning: https://arxiv.org/abs/2205.11916
Original comment LLMs are a 400-year-long confidence trick
LordDragonfang
Neutral mention · rule
"Thinking" is a term of art referring to the hidden/internal output of "reasoning" models where they output "chain of thought" before giving an answer[1]. This technique and name stem from the early observation that LLMs do better when explicitly told to "think step by step"[2]. Hope that helps clarify things for you for future constructive discussion. [1] https://arxiv.org/html/2410.10630v1 [2] https://arxiv.org/pdf/2205.11916
Original comment Magistral — the first reasoning model by Mistral AI
TeMPOraL
Neutral mention · rule
You're right about "living in the “overselling all up in your ears” epoch", but a good first defense against "being sold pffts as badabooms" is to blanket distrust all the marketing copy and whatever the salespeople say, and rely on your own understanding or experience. You may lose out on becoming an early adopter of some good things, but you'll also be spared wasting money on most garbage. With that in mind, I still don't get the dismissal. LLMs are broadly accessible - ever since the first ChatGPT, anyone could easily get access to a SOTA LLM and evaluate it for free; even the limited number of requests on free tiers were then, and now are, sufficient to throw your own…
Read the full comment
personal and professional problems at models and see how they do. Everyone can see for themselves this is not hot air - this is an unexpected technological breakthrough that's already overturning way people approach work, research and living, and it's not slowing down. I'd say: ignore what the companies are selling you - especially those who are just building products on top of LLMs and promising pie in the sky. At this point in time, they aren't doing anything you couldn't do for yourself with ChatGPT or Claude access[0]. We are also beginning to map out the possibilities - two years since the field exploded is very little time. So in short, anything a business does, you could hack yourself - and any speculative idea for AI applications you can imagine, there's likely some research team working on it too. The field is moving both absurdly fast and absurdly slow[1]. So your own personal experience over applying LLMs to your own problems, and watching people around you do the same, is really all you need to tell whether LLMs are hot air or not. My own perspective from doing that: it's not hot air. The layer of hype is thin, and in some areas the hype is downplaying the impact. -- [0] - Yes, obviously a bunch of full-time professionals are doing much more work than you or me over couple evenings of playing with ChatGPT. But they're building a marketable product, and 99% of work that goes into that is something you do not need to do, if you just want to replicate the core functionality for yourself. [1] - I mean, Anthropic just published a report on how exposing "thinking" capability to the model in form of a tool call leads to improvement of performance. On the one hand, kudos to them for testing this properly and publishing. On the other hand, that this was something to do was stupidly obvious ever since 1) OpenAI introduced function calling and 2) people figured out "Let's think step by step" improves model performance - which was back in 2022[2]. It's as clear example as ever that both hype and productization lag behind what anyone paying attention can do themselves at home. [2] - https://arxiv.org/abs/2205.11916
Original comment ChatGPT can't kill anything worth preserving
jdietrich
Neutral mention · rule
This isn't something I should convince you of. Just open up ChatGPT or Claude and try it for yourself. Think up a batch of your own questions and see how a modern LLM fares. I assure you that it'll do much better than chance. If you're so inclined, you can run enough tests to achieve statistical significance in the course of your lunch break. It depresses me that we seem to be spending more time arguing and hypothesising about LLMs than empirically testing them. The question of whether LLMs can think is completely settled, as their performance at zero-shot problems is simply impossible through pure memorisation or pattern-matching. The question that remains is far more interesting - how do they think? https://arxiv.org/pdf/2205.11916
Original comment Large models of what? Mistaking engineering achievements for linguistic agency
kromem
Neutral mention · rule
CoT doesn't require it to be a few shot prompt though. That was in the original paper, as you point out, but even something as simple as a zero shot prompt followed by "let's think step by step" has significant improvement: https://arxiv.org/abs/2205.11916 Do you still consider a zero-shot-CoT to be hinting? (And I don't mean the [0] example you shared - biasing results by continuing to ask for refinements in incorrect answers but not correct answers is an area where a number of papers have already had issues, but is not part of what I'm discussing - I mean in single prompt/response cycles). I agree that a consistently overlooked component of generators is their role as…
Read the full comment
classifiers, though this currently seems to be overlooked more outside of research circles as all too common has become using GPT-4 to evaluate a study results in place of manual review (a bit disconcerting). I'm not entirely sure how we ended up on diffusion and GANs - I'd meant transmission from transformer to transformer like Orca or the work that followed. Though it was an interesting read nonetheless and I appreciated your writing it.
Original comment Compare how GPT-2, 3, 3.5 and 4 answer the same questions
convexfunction
Neutral mention · rule
I'm not sure if "helpful answer" as opposed to "answer" makes much of a difference in answer quality -- I'd believe it helps a little, just don't know that it's been studied -- but a lot of silly stuff like that does definitely make a big difference in response quality on certain tasks. "Let's think step by step:" at the end of your prompt is probably the best-known one: https://arxiv.org/pdf/2205.11916.pdf
Original comment Re-implementing LangChain in 100 lines of code
andyk
Neutral mention · rule
My personal point was just to document my exploration of using prompts to generate new prompts where the generated prompts contain state that moves closer to an end goal (which in this case I compared to the concept of a base case in recursion). For whatever reason Patrick H. Winston's MIT OCW lecture on Cognitive Architectures always stuck with me, and in particular his summary of the historical system from CMU called General Problem Solver (GPS) in which they try to identify a goal and then have the AI evaluate the difference between the current state and the goal and try take steps to bridge the gap. https://www.youtube.com/watch?v=PimSbFGrwXM&t=189s The ability for…
Read the full comment
LLMs to break down problem into sub-steps (a la "Let's think step by step" [1]) reminded me of this part of Winston's lecture. And so I wanted to try making a prompt that (1) contains state and (2) can be used to generate another prompt which has updated state. [1] Large Language Models are Zero-Shot Reasoners - https://arxiv.org/abs/2205.11916
Original comment Show HN: Recursive LLM Prompts
vsareto
Neutral mention · rule
>Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs https://arxiv.org/abs/2205.11916 >https://sites.google.com/view/automatic-prompt-engineer Not exactly a "toaster go brrrr" job, but it could be obsolete one day WaPo does need to chill though. There’s barely any Prompt Engineer jobs
Original comment Tech’s hottest new job: AI whisperer. No coding required
simonw
Neutral mention · rule
My personal excitement about language models is based on what they can do today. I'm a big believer in the "capability overhang" idea, which is that the existing language models still have a huge array of capabilities that we haven't discovered yet. That theory seems to be proved correct on a constant basis. Even the classic "let's think about this step by step" paper came out less than a year ago: https://arxiv.org/abs/2205.11916 - May 2022.
Original comment ChatGPT Cheat Sheet
Isinlor
Neutral mention · rule
Small disclaimer: I don't want to take position on sentience or consciousness. "LaMDA is not an agent with desires that it expresses in language." Some counter points: - being an agent is not a high bar to clear; a thermostat is an agent - "with desires" - LaMDA can be thought of as rational agent whose objective function is to produce text with high likelihood - "that it expresses in language" it expresses likelihood of next token in language "it can't fear anything because there is no time for it to do so." There is a time for it to experience something and that is when it does feed forward pass. Also, as you correctly notices, LaMDA is not only doing greedy text…
Read the full comment
generation, but also a deeper text completion search. But in fact just prompting a model like GPT-3 or Gopher with "let's think step by step" and letting it process in context is already significantly improving performance on wide variety of reasoning tasks [0]. I don't see how this is functionally different from human pondering. The only objection that seems reasonable to me is that LaMDA is not correctly describing it's internal state, because it will happily generate descriptions of its internal state that we know are physically impossible. Like being a squirrel. But I don't see how you can describe a system that will be fundamentally functionally more capable than closed-loop computation. LaMDA is almost certainly Turing complete. Postulating that it is fundamentally not capable of doing something is the same as rejecting Church-Turing thesis and postulating super-Turing computational model. That's a heavy claim. [0] https://arxiv.org/abs/2205.11916
Original comment LaMDA is not sentient
TeMPOraL
Neutral mention · rule
I appreciate you giving solid overview and references of the use and context of that term - however, in this case, I fear that ship has long sailed by now. The more relevant paper would be[0], the title of which is, I believe, the source of the "LLMs are zero-shot [learners / reasoners / whatever]" phrase that's been reaching meme status, and now shows up in training materials. -- [0] - https://arxiv.org/abs/2205.11916 - "Large Language Models are Zero-Shot Reasoners"
Original comment What we learned from using GPT for 500k+ classifications

Keep following the thread.

More in Language models

More from the same topics.