AI models collapse when trained on recursively generated data

> while it's an open question I think theres research that shows that models collapse after being trained on their own content. https://www.nature.com/articles/s41586-024-07566-y

I don't see why this wouldn't just lead to model collapse: https://www.nature.com/articles/s41586-024-07566-y If you've spent any time using LLMs to write documentation you'll see this for yourself: the compounding will just be rewriting valid information with less terse information. I find it concerning Karpathy doesn't see this. But I'm not surprised, because AI maximalists seem to find it really difficult to be... "normal"? Rule of thumb: if you find yourself needing to broadcast the special LLM sauce you came up with instead of what it helped you produce, ask yourself why.

There's a limit to that according to: https://www.nature.com/articles/s41586-024-07566-y . Basically, if you use an LLM to augment a training dataset it will become "dumber" every subsequent generation and I am not sure how you can generate synthetic data for a language model without using a language model

I do writing with RAG and it can be implemented to suprisingly good if you already have your own writing that the text is being generated from. FAQs etc can be pretty easy when your content is context for the AI. After a few rounds of AI generating AI content from AI content, I'm sure it could eventually become slop...like the model collapse lol idk. "AI models collapse when trained on recursively generated data" - https://www.nature.com/articles/s41586-024-07566-y

But, the internet is already polluted and being constantly polluted by LLMs. It feels like LLMs are the PFAS of the internet. So, I'm not sure what your argument is, but I would welcome clarification. Maybe I am misunderstanding your definition of model collapse. The article references this definition of model collapse[0]. I'm curious when you say "that's not what model collapse is" -- what do you mean by model collapse that is different from the definition of the article and reference? [0] https://www.nature.com/articles/s41586-024-07566-y

> But as it started consuming more and more data that it had produced itself, its reliability became close to none. It's exactly the opposite with LLMs. See the "model collapse" phenomenon (https://www.nature.com/articles/s41586-024-07566-y). > We show that, over time, models start losing information about the true distribution, which first starts with tails disappearing, and learned behaviours converge over the generations to a point estimate with very small variance. Furthermore, we show that this process is inevitable, even for cases with almost ideal conditions for long-term learning, that is, no function estimation error

> I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs. At the extreme, there is this paper on ‘inbred LLMs’: https://www.nature.com/articles/s41586-024-07566-y

Synthetic data without some kind of external validation is garbage. E.g. you can't just synthetically generate code, something or someone needs to run it and see if it performs the functions you actually asked of it. You need to feed the LLM output into some kind of formal verification system, and only then add it back to the synthetic training dataset. Here, for example - dumb recursive training causes model collapse: https://www.nature.com/articles/s41586-024-07566-y

This has been one [unintended?] consequence of AI promulgation. A direct disincentive toward the kind of open access that so used to be common and provided a lot of low-hanging fruit for independent developers trying to increase interoperability within their favorite niches. So now not only is AI filling the web with garbage that poisons future model development[1], it provides incentive to further close and wall off access to (user-provided!) data. 1. https://www.nature.com/articles/s41586-024-07566-y