Skip to content
The methodOpen data
Back to the collection

Large Language Models Can Self-Improve

Jiaxin Huang, Shixiang Shane Gu, Le Hou

Show all 7 authors

Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han

2022 · arXiv

Language modelsDistributed systems
Read the paper
HN score
0
Mentions
7
Readers
7
Months
4
What’s in a score?

From the abstract

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

taneq
Neutral mention · rule
> This result feels like creepy perpetual motion. It’s like they’re teaching the AI ethics by making it write an ethics textbook and then read the textbook it just wrote. Is this a free lunch? Shouldn’t it be impossible for the AI to teach itself any more ethics than it started out with? Isn't this kind of feedback loop precisely what's predicted at the heart of the Singularity? A machine learning system reaching the point where it's able to improve itself without further human guidance? "Large Language Models Can Self-Improve" (https://arxiv.org/abs/2210.11610) describes something similar, increasing an LLM's performance by fine tuning it on its own 'high confidence'…
Read the full comment
answers. I'd expect there to be a limit to the amount of extra improvement possible using this method, the question is whether this improvement is enough to push the system past the performance of the researchers that created it. (As an aside, is there a theoretical limit to how intelligent a system can be for a given quantity of training data? Of course, we'd need a rigorous definition of 'intelligence' and 'data'...)
Original comment Constitutional AI: RLHF on Steroids
kzrdude
Neutral mention · rule
I don't think it can be so easily dismissed. AlphaZero contributed lots, for example reproducing opening theory from a blank slate. For chess that's significant. But we don't care about chess for this discussion about GPT. Self-learning and self-play would be interesting because it's an amplification factor. Some searching suggests some researchers have been thinking along these lines. Maybe this? https://arxiv.org/abs/2210.11610 Going for methods that don't require supervision
Original comment GPT-3 will ignore tools when it disagrees with them
jbenjoseph
Neutral mention · rule
I have seen no evidence for that, only the opposite: https://arxiv.org/abs/2210.11610 Intuitively, training a simple enough linear statistical model with its own output should be a NOP. But LLMs are anything but simple models, so I think the non-linearities may be synthesizing new useful information. Similarly to how all of maths can be synthesized from a few basic axioms with enough intelligence or computation.
Original comment Eventually ChatGPT will eat itself
valine
Neutral mention · rule
There is evidence to suggest that LLMs have recently passed a threshold where their performance can be improved by training on their own output. https://arxiv.org/abs/2210.11610 I expect your concerns are likely unwarranted.
Original comment Are we racing toward AI catastrophe?
anabis
Neutral mention · rule
>But, it's not implausible that some future model will simply have a superhuman learning rate and a superhuman ability to distinguish "right" from "wrong" - this paragraph will look downright prophetic then. There is already a paper for that: https://arxiv.org/abs/2210.11610 Large Language Models Can Self-Improve >Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a…
Read the full comment
pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.
Original comment ChatGPT Is a Blurry JPEG of the Web
visarga
Neutral mention · rule
It was not a language model, but AlphaZero played against itself millions of games and trained on the data it generated, without seeing any human games. And it reached super human level, of course. Why was this possible? The game itself acted as an anti-bullshit filter. So you can train on your own generated data if you filter it. Like this one: Large Language Models Can Self Improve https://arxiv.org/abs/2210.11610
Original comment ChatGPT Can't Kill Anything Worth Preserving

Keep following the thread.

More in Language models

More from the same topics.