Skip to content
The methodOpen data
Back to the collection

Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

Eric Zelikman, Georges Harik, Yijia Shao

Show all 6 authors

Varuna Jayasiri, Nick Haber, Noah D. Goodman

2024 · arXiv

Language modelsNetworksMachine learning
Read the paper
HN score
0
Mentions
6
Readers
6
Months
5
What’s in a score?

From the abstract

When writing and talking, people sometimes pause to think. Although reasoning-focused works have often framed reasoning as a method of answering questions or completing agentic tasks, reasoning is implicit in almost all written text. For example, this applies to the steps not stated between the lines of a proof or to the theory of mind underlying a conversation. In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting -- ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. We address key challenges, including 1) the computational cost of generating continuations, 2) the fact that the LM does not initially know how to generate or use internal thoughts, and 3) the need to predict beyond individual next tokens. To resolve these, we propose a tokenwise parallel sampling algorithm, using learnable tokens indicating a thought's start and end, and an extended teacher-forcing technique. Encouragingly, generated rationales disproportionately help model difficult-to-predict tokens and improve the LM's ability to directly answer difficult questions. In particular, after continued pretraining of an LM on a corpus of internet text with Quiet-STaR, we find zero-shot improvements on GSM8K (5.9%$\rightarrow$10.9%) and CommonsenseQA (36.3%$\rightarrow$47.2%) and observe a perplexity improvement of difficult tokens in natural text. Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR marks a step towards LMs that can learn to reason in a more general and scalable way.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

ValentinA23
Neutral mention · rule
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking https://arxiv.org/pdf/2403.09629 > In the Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned by inferring rationales from few-shot examples in question-answering and learning from those that lead to a correct answer. This is a highly constrained setting – ideally, a language model could instead learn to infer unstated rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR in which LMs learn to generate rationales at each token to explain future text, improving their predictions. >[...] >We generate thoughts, in parallel, following all tokens in the…
Read the full comment
text (think). The model produces a mixture of its next-token predictions with and without a thought (talk). We apply REINFORCE, as in STaR, to increase the likelihood of thoughts that help the model predict future text while discarding thoughts that make the future text less likely (learn).
Original comment g1: Using Llama-3.1 70B on Groq to create o1-like reasoning chains
FeepingCreature
Neutral mention · rule
Saying that LLMs should be able to do problems without CoT is a bit like saying that humans should be able to write programs without thinking though. CoT is fundamentally an architectural necessity owed to the finite cognitive effort possible per token. Let it be CoT, let it be QuietSTaR ( https://arxiv.org/abs/2403.09629 ), let it be pause tokens ( https://arxiv.org/abs/2310.02226 ), let it even be dots ( https://arxiv.org/abs/2404.15758 ), but the thinking has to happen somewhere. I think mostly, LLMs at the moment are incredibly uneven. LLM assistants can pull obscure knowledge out of nowhere one second and fail extremely basic reasoning the next. So just because some example happens a lot in the source material doesn't mean the LLM can learn it. IMO, that CoT works at all is more down to luck of the training set than any inherent capability of the LLM.
Original comment The "Chain of Thought" Delusions
rfdearborn
Neutral mention · rule
The trendline is definitely toward increasing dynamic routing, but I suspect it's more so that MoE/MoD/MoDE enable models to embed additional facts with less superposition within their weights than enable deeper reasoning. Instead I expect deeper reasoning will come through token-wise dynamism rather than layer-wise -- e.g., this recent Quiet-STaR paper in which the model outputs throwaway rationale tokens: https://arxiv.org/abs/2403.09629
Original comment Mixture-of-Depths: Dynamically allocating compute in transformers
snats
Neutral mention · rule
It's been pretty interesting to see some research trying to solve this. For example, Stanford researchers recently published QuietSTaR[1]. This makes LLMs "think" before they speak. By generating a chain of thoughts and just choosing the best possible path from the generated thoughts. This method is better than Chain of Thought and I think is a step in the right direction. [1] https://arxiv.org/abs/2403.09629
Original comment Can Demis Hassabis save Google?
visarga
Neutral mention · rule
> Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking This paper suggests that a large language model should "think ahead" by predicting not only the next token but also a "supporting thought." The approach involves generating all tokens simultaneously, allowing for a single forward pass that produces both the next token and a supporting thought, which might consist of, for example, 16 tokens. This supporting thought influences the model's prediction. The process is then extended to multiple supporting thoughts by ingeniously masking cross-attention between thoughts to ensure their independence. So in essence we can fill all the remaining context…
Read the full comment
with supporting thoughts and benefit from all of them in the same single forward pass. The supporting thoughts themselves are trained with the objective to maximize the probability of a longer sequence ahead, using RL. So they are trained to optimize for longer-term, instead of the myopic next token prediction task. https://arxiv.org/abs/2403.09629
Original comment How Chain-of-Thought Reasoning Helps Neural Networks Compute

Keep following the thread.

More in Language models

More from the same topics.