Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT introduced such language models to the general public. It is now clear that large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images. In this paper we consider what the future might hold. What will happen to GPT-{n} once LLMs contribute much of the language found online? We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs. We build theoretical intuition behind the phenomenon and portray its ubiquity amongst all learned generative models. We demonstrate that it has to be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of content generated by LLMs in data crawled from the Internet.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
You're confusing it with data poisoning.
Model collapse itself is(was?) a fairly serious research topic: https://arxiv.org/abs/2305.17493
We've by now reached a "probably not inevitable" - https://arxiv.org/abs/2404.01413 argues there's a finite upper bound to error - but I'd also point out that that paper assumes training data cardinality increases with the number of training generations and is strictly accumulative.
To a first order, that means you better have a pre-2022 dataset to get started, and have archived it well.
but it's probably fair to say current SOTA is still more or less "it's neither impossible nor inevitable".
https://arxiv.org/pdf/2305.17493
There's some cursory indication that in the long tail, training LLMs on LLM-generated data causes model collapse. Kind of like how if you photocopy a photocopy too many times the document becomes unreadable.
This isn't really surprising though. Neural networks at large are a form of lossy compression. You can't do lossy compression on artifacts recovered from lossy compression too many times. The losses stack.
> Paradoxically, I think a lot of research is showing that synthetic training information can be just as good as the real stuff.
Which studies show this? https://arxiv.org/abs/2305.17493 shows the exact opposite and my (layman's) understanding of statistics and epistemology lines up entirely with this finding.
Like, how could this even theoretically work? In the best case scenario wouldn't training on synthetic training data make LLMs overconfident / overfit the data once faced with new (human) input to respond to?
> For better or worse, the quality of output LLMs can produce based on such publicly available data appears to be peaking. Recent work has documented that GPT, for example, has begun to exhibit declines in the quality of its output.3
It suggests no such thing, and the problems with citation 3 (https://arxiv.org/abs/2307.09009) have been extensively discussed.
> It has been suggested this decline in performance is the expected result of a feedback loop, wherein data collected for training is increasingly contaminated by GPT itself, as users leverage the technology to produce and post content online.12
And the reasons why #12 (https://arxiv.org/abs/2305.17493) are totally inapplicable to the real world, as opposed to their artificial toy model where no kind of curation or selection whatsoever is done as synthetic samples approach 100% of the training data, have also been extensively discussed.
> AI used to be weird. Now ‘sounds like a bot’ is just shorthand for boring.
It's real, real hard to escape regression to the mean in autonomous systems. That gaussian's gonna get ya.
[1] I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson, “The Curse of Recursion: Training on Generated Data Makes Models Forget,” May 2023, [Online]. Available: http://arxiv.org/abs/2305.17493
One of the world's leading experts on information security systems, Professor Ross Anderson of Cambridge University, in his article https://arxiv.org/abs/2305.17493v2 is already considering a scenario where most of the Internet will be a neural network-generated hallucination. But neither you, nor me, nor any expert or anyone at all will be able to distinguish between fake and real. As a result, neural networks will degenerate in the process of learning from their own hallucinations of the Internet, and humans will degenerate in the process of applying the degenerated neural networks. This process he called "neural network collapse". It can only be countered by a…
Read the full comment
collective neural network of all the minds of humanity. For mutual validation and self-improvement of LLM and humans, we need the ability to match the knowledge of artificial intelligence with collective intelligence. This is what can get us out of the personal reality tunnels and personal information bubbles in which we are getting deeper and deeper stuck individually.
This network of real human knowledge provides a way to introduce new, clean, human-generated datasets into LLM training in a validity-conscious manner, and makes it possible to avoid model collapse and reduce unwanted errors in creating new and better generations of generative models. And we have a practical solution to avoid the collapse of large language models — we create a global unbiased decentralized CyberPravda (dot) com platform for disputes, for analyzing the reliability of information and assessing the reputation of its authors, where people are accountable with personal reputation for their knowledge and arguments.
maybe. A lot of modern clickwork is actually model training and there is a model-collapse phenomena (https://arxiv.org/abs/2305.17493) which means that it should be banned for such work. I bet a number of clever people on the platforms are already trying to instrument AI to do the work regardless - it's pretty close to "free money" if you can pull it off and not get caught.
>> There's already been the studies that these models quickly break down when they're fed their own generated data as training data
Not "these models" if you're referring to large language models pretrained and fine-tuned with Transformers. The only study I am aware of that explored the effect you discuss [1] had to make do with very different models as proxies: Variational Autoencoders, Mixture of Gaussians and a smaller LLM with 125 million parameters that hardly qualifies as one. It is not safe to extrapolate those results to really Large-LMs trained with Transformers, as it is not safe to extrapolate any results on one neural net architecture on another.
Note the…
Read the full comment
Mixtures of Gaussians aren't even neural nets at all, but a completely different statistical model.
The question of what will happen once a few generations of LLMs have trained on their ancestors' outputs is an interesting one that has not yet been clearly answered. In the long run, future developments may even mean it's not really a valid question at all because such feedback retraining can't happen in practice- who knows?
I for one remain curious to see how this goes.
______________________
[1] The Curse of Recursion: Training on Generated Data Makes Models Forget https://arxiv.org/abs/2305.17493
Fundamentally untrue, and disheartening that it's the top comment.
You can't use a model's output to train another model, it leads to complete gibberish (termed "model collapse"). https://arxiv.org/abs/2305.17493v2
And the Llama 2 license allows users to train derivative models, which is what people really care about. https://github.com/facebookresearch/llama/blob/main/LICENSE
That paper[0] mentions two defects, catastrophic forgetting which occurs during continual learning. GPT type models do not use this type of training and use supervised training instead which utilizes a curated corpus of text. The second is model collapse which is essentially just bad data making its way into the training set, once again a sufficiently curated corpus eliminates this problem.
Microsoft has published a few papers about their research in using LLMs to train new LLM models. They have used training data consisting of essentially recorded chatGPT interactions which consist of the system prompt, user request and model response[1] to successfully train a smaller…
Read the full comment
model that outperforms Vicuna-13B. The other paper[2] demonstrates using a combination of curated web data and machine generated textbooks and exercises (using GPT-3.5) and they produced a small model (1.3B) that outperforms several larger models on programming tasks.
The point here is that the simple presence of machine generated content in the dataset is not an immediate detriment to the model being trained on it. It all hinges on the quality of the data regardless of whether it is of human or machine origin. Think about it, if the information is correct then there is no difference whether it came from a human or a machine. Machine generated data is not secretly cursed or other such nonsense. It's either correct or incorrect, nothing more.
[0] https://arxiv.org/pdf/2305.17493v2.pdf
[1] https://arxiv.org/pdf/2306.02707.pdf
[2] https://arxiv.org/pdf/2306.11644.pdf
Related: https://arxiv.org/abs/2305.17493v2
Basically, if this paper is right, 2021 was the last year usable training data could be pulled from the internet. It could be all downhill from hereon out, as the amount of AI generated content will soon overtake actual human generated and -curated data on the internet.
So the party could be over soon ¯\_(ツ)_/¯
Neural networks will degenerate in the process of learning from their own hallucinations, and humans will degenerate in the process of applying the degenerated neural networks. This process is called "neural network collapse". https://arxiv.org/abs/2305.17493v2 It can only be countered by a collective neural network of all the minds of humanity. For mutual validation and self-improvement of LLM and humans, we need the ability to match the knowledge of artificial intelligence with collective intelligence. Only the CyberPravda project is the practical solution to avoid the collapse of large language models.
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
In honor of Alan Turing's hundredth birthday, I unwisely set out some thoughts about one of Turing's obsessions throughout his life, the question of physics and free will.