Skip to content
The methodOpen data
Back to the collection

Training Compute-Optimal Large Language Models

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch

Show all 22 authors

Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre

2022 · arXiv

Language modelsStatistics & probability
Read the paper
HN score
0
Mentions
31
Readers
27
Months
16
What’s in a score?

From the abstract

We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 31 total mentions

namr2000
Neutral mention · rule
You don't have to train from scratch but you can. Distillation ends up being somewhere in the ballpark of 1000x faster to train [1]. It also comes with the huge advantage of not needing to create RLHF datasets, since you can just copy the behavior of the teacher model. This saves an enormous amount of labeling money at the cost of making the model behave similarly to the teacher. If you are training from scratch, you can look at LLM scaling laws to figure out roughly the compute budget you need to optimally train a model [2]. Based on [2] a 30B model needs something like 2e+23 FLOPS to train from scratch whereas a 1.6T model needs something like 1e+27 FLOPs to train. So…
Read the full comment
DeepSeek v4 Pro was roughly 5000x more expensive to train than this model. I'm not totally sure how MOE affects scaling laws, so these numbers might be different in reality, but it gives you a good ballpark estimate of the difference in training scale. [1] https://arxiv.org/abs/2505.12781 [2] https://arxiv.org/abs/2203.15556
Original comment Cohere's First Model for Developers
versteegen
Neutral mention · rule
Because the training data/model size/compute tradeoff derived from that paper is highly suboptimal (too many parameters) compared to the ones from the later Deepmind scaling laws [1]. And then Meta researchers recommended using even smaller models, to trade-off training- and inference-time compute [2] (which I thought was pretty obvious if you care about more than just benchmarks). [1] https://arxiv.org/abs/2203.15556 Training Compute-Optimal Large Language Models [2] https://arxiv.org/abs/2302.13971 LLaMA: Open and Efficient Foundation Language Models
Original comment Do large language models need all those layers?
boomanaiden154
Neutral mention · rule
One of the biggest things that seems to be holding back ML in compilers right now is dataset size. This model was only trained on a gigabyte of source code, 30+% of that synthetic. Even on much simpler models, there have been massive performance gains by just throwing more data at them. Some experimentation with the original MLGO inlining model on a much bigger data corpus doubled the code-size wins. LLMs have also been shown to perform better they more data they are fed [1]. We have a solution for this and are hoping to put something out in the next week. 1. https://arxiv.org/abs/2203.15556
Original comment Large Language Models for Compiler Optimization
airgapstopgap
Neutral mention · rule
Do you realize that LLaMA-1 is just a very slightly smaller, comparably performing replication of Chinchilla [1], which DeepMind had completed a year prior to LLaMA's release? And has RLHF-ed into a suitable chatbot "Sparrow" [2] months earlier than ChatGPT was launched? To assume that Google doesn't have anything competitive with Meta is to say that their papers just so happen to contain recipes for Meta's models but they've arrived at those not through training and benchmarking but by divination and bullshitting. This, let us say, does not sound plausible. Then again, Microsoft uses LLaMA for research, and they should theoretically have some ability to get stuff from OpenAI. Evidently this isn't how any of this works, huh. 1. https://arxiv.org/abs/2203.15556 2. https://en.wikipedia.org/wiki/Sparrow_(bot)
Original comment Llama 2 – Meta AI
gdiamos
Neutral mention · rule
This initial public alpha release of Lamini LLM fine tuning includes several advanced optimizations: 1. Chinchilla recipe smaller models pretrained on data increases inference speed 2. Instruction fine tuning training on a small high quality set of instructions unlocks the knowledge learned during foundation model training. 3. Latency constrained batching achieves high utilization under load during token generation 4. Containerized SLURM combines fast scheduling of SLURM with LLM containers 5. Mixed precision training uses matrix operations for training There are so many low hanging fruits in LLM tuning, steering, and alignment. We are just getting started on this for…
Read the full comment
enterprise and open source. For this reason I disagree with Sam Altman that the age of bigger models is over. We are still leaving orders of magnitude on the table, e.g. by not including optimizations like sparsity in these models. References for inspiration: [1] - https://arxiv.org/abs/2203.15556 [2] - https://arxiv.org/abs/1910.10683 [3] - https://www.usenix.org/system/files/osdi22-yu.pdf [4] - https://www.schedmd.com/ [5] - https://arxiv.org/abs/1710.03740
Original comment Free and Fast LLM Finetuning
thewataccount
Neutral mention · rule
It's from this paper - https://arxiv.org/abs/2203.15556 Like the sibling comment said - the proportion of training tokens to parameter size is very important, and there's a certain threshold needed to be met for it to be "fully trained". Usually you have a fixed amount of compute (budget/time essentially) - and in that case you want to pick the largest parameter count that you can fully train, and not the largest parameter count your hardware can support and then train that for less time. tl;dr - Small models with training over the chinchilla threshold can out perform large models that are undertrained EDIT: Figure 2 page 5, and Table 3 page 8 - might be worth checking out.
Original comment SparseGPT: Language Models Can Be Accurately Pruned in One-Shot
ftxbro
Neutral mention · rule
Chinchilla scaling laws aren't necessarily used for the tradeoffs of training runs like this one. Here is the goal of Chinchilla scaling from https://arxiv.org/pdf/2203.15556.pdf : "In this work, we revisit the question: Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens? To answer this question, we model the final pre-training loss L(N, D) as a function of the number of model parameters N, and the number of training tokens, D. Since the computational budget C is a deterministic function FLOPs(N, D) of the number of seen training tokens and model parameters, we are interested in minimizing L under the constraint FLOPs(N, D) =…
Read the full comment
C." But this isn't the only interesting optimization. It can also be useful to have small models that are "overtrained" relative to chinchilla optimality. There are practical reasons to prefer smaller models over big models even at some cost of training expense or pre-training loss, but the chinchilla optimization does not account for this at all. For one thing, the smaller models can fit in the memory of a wider class of devices. For another thing, smaller models incur less computational expense at inference time. Yet another reason is that they will probably have less latency.
Original comment Llama 1.3B Trained on 200B Tokens for Commercial Use
crakenzak
Neutral mention · rule
I agree with your general statement, but if one thing the recent DeepMind Chinchilla research paper showed us is that the size of the model (number of parameters) is much less of the determinant of model performance & quality than the amount of high-quality data (number of high-quality tokens). Their 70B chinchilla model significantly outperforms the 175B GPT3 model. Possibly where OpenAI has a leg up is their high-quality data sourcing & curating infrastructure and their RLHF mechanisms. Paper: https://arxiv.org/abs/2203.15556
Original comment The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT
jacobn
Neutral mention · rule
Both matter, and returns fall off as you go further in one but not the other. The Chinchilla paper[0] established a simple scaling law for Large Language Models: model size and training tokens should grow at the same pace. Compute is then proportional to the product of model size & data quantity. That said, quality of data also matters a lot - OpenAI has had human labelers produce the data for their Reinforcement Learning from Human Feedback (RLHF), which has probably had a disproportionate impact on the success of ChatGPT compared to previous models, but that data is probably O(1%) of what they trained on. At this point I'm guessing OpenAI are limited by both data & compute. Rumor has it they're training the "next big thing" now and it won't finish until December. If they had more compute they could presumably finish sooner, and if they had more data they would presumably let it train longer. [0] https://arxiv.org/abs/2203.15556
Original comment The Prospect of an AI Winter
DavidSJ
Neutral mention · rule
GPT 3.5 likely differs from the original GPT 3 by more than instruction fine-tuning. For example, it was probably retrained under Chinchilla scaling laws [1], with a lot more data and maybe a somewhat smaller parameter count. There are many variants of GPT-3 and GPT-3.5, so which one Meta is comparing against makes a big difference. [1] https://arxiv.org/abs/2203.15556
Original comment LLaMA-7B in Pure C++ with full Apple Silicon support
code_runner
Neutral mention · rule
I’ll do my best. Number of params is the number of weights. Basically the number of learnable variables. Number of tokens is how many tokens it saw during training. Vocab size is the number of distinct tokens. The relationship between params/tokens/compute power is something people have studied a good deal and how it affects model performance. https://arxiv.org/pdf/2203.15556.pdf
Original comment GPT in 60 Lines of NumPy
sbierwagen
Neutral mention · rule
If the number of bits in the net is smaller than the number of bits in the training data, then training is always going to be a "lossy" operation, losing data in the strictest sense. So from that perspective, the source text is "more" "valuable". Training is a fairly mechanical operation-- you lay out the architecture, decide on an encoding/weighting scheme, then feed the training data through the net. Not a lot of secret sauce there. (As evidenced in the explosion in competing LLMs after the GPT papers were published) Going forward, the copyright status and the possible need to pay licensing fees when training a neural net on (say) a book will be hot litigation topic.…
Read the full comment
Before 2019 this just was not a thing at all. ("Fair use" covered anything done to a book in your possession. After a library buys a book, do they pay an additional fee every time a patron opens it?) Read "Training Compute-Optimal Large Language Models" paper for some mediations on the tradeoff between model size and training data volume: https://arxiv.org/pdf/2203.15556.pdf
Original comment Ask HN: What's the value ratio of GPT3's LLM vs. its training data?

Keep following the thread.

More in Language models

More from the same topics.