We investigate the potential constraints on LLM scaling posed by the availability of public human-generated text data. We forecast the growing demand for training data based on current trends and estimate the total stock of public human text data. Our findings indicate that if current LLM development trends continue, models will be trained on datasets roughly equal in size to the available stock of public human text data between 2026 and 2032, or slightly earlier if models are overtrained. We explore how progress in language modeling can continue when human-generated text datasets cannot be scaled any further. We argue that synthetic data generation, transfer learning from data-rich domains, and data efficiency improvements might support further progress.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
> Where are they going to get more data?
This is a great question, but note that folks were freaking out about this a year or so ago and we seem to be doing fine.
We seem to be making progress with some combination of synthetic training datasets on coding/math tasks, textbooks authored by paid experts, and new tokens (plus preference signals) generated by users of the LLM systems.
It wouldn’t surprise me if coding/math turned out to have a dense-enough loss-landscape to produce enough synthetic data to get to AGI - though I wouldn’t bet on this as a highly likely outcome.
I have been wanting to read/do some more rigorous analysis here though.
This sort of analysis would count as the kind of rigorous prediction that I’m asking for above.
E2A: initial exploration on this: https://chatgpt.com/share/68d96124-a6f4-8006-8a87-bfa7ee4ea3...
Gives some relevant papers such as
https://arxiv.org/html/2211.04325v2#:~:text=3.1%20AI
I have no doubt that this "feature" is a backroom deal worth millions because OpenAI is running out of public internet data with which to improve its models. (See this paper from researchers at MIT and a few other schools which predicts that high-quality text training data will be 'used up' by 2026: https://arxiv.org/abs/2211.04325 )
Think of all of the email, Google Docs, and other data that Alphabet has that it can use to train and improve its models. OpenAI has limited ways to get non-public text data unless Microsoft is giving them some data from Office users, Hotmail users.
Just my two cents. And whatever Dropbox is doing with retrieval augmented generation (RAG) / "new+better search" with the OpenAI APIs: I'm certain it could be done with less latency and probably would cost less if the RAG 'feature' / 'new search' was built in house at Dropbox.
there's a paper that basically shows these LLMs are pretty much tapped out in terms of quality data to train on, not sure if there are any legit architecture improvements in the works to improve results quality but it seems brute force training with more data may be about over - https://arxiv.org/pdf/2211.04325.pdf
As for SEO, Google is going to have to stop being lazy and relying so much on backlinks and domain authority. They might have to consider allowing searchers to essentially vote on what they consider good. I don't see any other way low quality AI content doesn't dominate SERPs, established players will spin up entire sites and spam backlinks to them all. There are also people working on using AI to identify obvious AI content by finding patterns in the output which is probably our only hope.
For social media and forums they will have to rely even harder on captcha or they will be taken over by bots talking to other bots
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.