Skip to content
The methodOpen data
Back to the collection

Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan

Show all 10 authors

Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei

2020 · arXiv

Language modelsStatistics & probability
Read the paper
HN score
0
Mentions
31
Readers
27
Months
27
What’s in a score?

From the abstract

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 31 total mentions

cute_boi
Neutral mention · rule
No need stupid moving texts. CS231n: Convolutional Neural Networks for Visual Recognition - https://cs231n.github.io/ The Unreasonable Effectiveness of Recurrent Neural Networks - https://karpathy.github.io/2015/05/21/rnn-effectiveness/ Understanding LSTM Networks - https://colah.github.io/posts/2015-08-Understanding-LSTMs/ ImageNet Classification with Deep Convolutional Neural Networks - https://papers.nips.cc/paper/2012/hash/c399862d3b9d6b76c8436... Deep Residual Learning for Image Recognition - https://arxiv.org/abs/1512.03385 Multi-Scale Context Aggregation by Dilated Convolutions - https://arxiv.org/abs/1511.07122 Identity Mappings in Deep Residual Networks -…
Read the full comment
https://arxiv.org/abs/1603.05027 Recurrent Neural Network Regularization - https://arxiv.org/abs/1409.2329 Deep Speech 2: End-to-End Speech Recognition in English and Mandarin - https://arxiv.org/abs/1512.02595 Order Matters: Sequence to Sequence for Sets - https://arxiv.org/abs/1511.06391 Neural Machine Translation by Jointly Learning to Align and Translate - https://arxiv.org/abs/1409.0473 Pointer Networks - https://arxiv.org/abs/1506.03134 Attention Is All You Need - https://arxiv.org/abs/1706.03762 The Annotated Transformer - https://nlp.seas.harvard.edu/annotated-transformer/ Neural Turing Machines - https://arxiv.org/abs/1410.5401 A Simple Neural Network Module for Relational Reasoning - https://arxiv.org/abs/1706.01427 Relational Recurrent Neural Networks - https://arxiv.org/abs/1806.01822 Neural Message Passing for Quantum Chemistry - https://arxiv.org/abs/1704.01212 Scaling Laws for Neural Language Models - https://arxiv.org/abs/2001.08361 GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism - https://arxiv.org/abs/1811.06965 Keeping Neural Networks Simple by Minimizing the Description Length of the Weights - https://www.cs.toronto.edu/~hinton/absps/colt93.pdf A Tutorial Introduction to the Minimum Description Length Principle - https://arxiv.org/abs/math/0406077 The First Law of Complexodynamics - https://scottaaronson.blog/?p=762 Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton - https://arxiv.org/abs/1405.6903 Kolmogorov Complexity - https://onlinelibrary.wiley.com/doi/book/10.1002/047174882X Variational Lossy Autoencoder - https://arxiv.org/abs/1611.02731 Machine Super Intelligence - https://www.vetta.org/documents/Machine_Super_Intelligence.p...
Original comment 30papers.com – Ilya's 30 essential ML papers, in a beginner friendly format
an0malous
Neutral mention · rule
No, well known in the current multiverse branch where we still occasionally use things like math and scientific analysis instead of people’s vibe checks and pelican SVGs. Here’s the paper from OpenAI where Dario himself was a co-author: https://arxiv.org/pdf/2001.08361 > We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N, dataset size D, and optimized training computation Cmin, as encapsulated in Equations (1.5) and (1.6). Conversely, we find very weak dependence on many architectural and optimization hyperparameters. Since scalings with N,D,Cmin are power-laws, there are diminishing returns with increasing scale.
Original comment GPT-5.5 hallucinates 3x more than MIT-licensed GLM-5.2
hodgehog11
Neutral mention · rule
Fine, that's fair. I believe the statement that you made is countered by my claim, which is: Theorem. For any tolerance epsilon > 0, there exists a transformer neural network of sufficient size that follows, up to the factor epsilon, the policy that most optimally achieves arbitrary goals in arbitrary stochastic environments. Proof (sketch). For any stochastic environment with a given goal, there exists a model that maximizes expected return under this goal (not necessarily unique, but it exists). From Solomonoff's convergence theorem (Theorem 3.19 in [1]), Bayes-optimal predictors under the universal Kolmogorov prior converge with increasing context to this model.…
Read the full comment
Consequently, there exists an agent (called the AIXI agent) that is Pareto-optimal for arbitrary goals (Theorem 5.23 in [1]). This agent is a sequence-to-sequence map with some mild regularity, and satisfies the conditions of Theorem 3 in [2]. From this universal approximation theorem (itself proven in Appendices B and C in [2]), there exists a transformer neural network of a sufficient size that replicates the AIXI agent up to the factor epsilon. This is effectively the argument made in [3], although I'm not fond of their presentation. Now, practitioners still cry foul because existence doesn't guarantee a procedure to find this particular architecture (this is the constructive bit). This is where the neural scaling law comes in. The trick is to work with a linearization of the network, called the neural tangent kernel; it's existence is guaranteed from Theorem 7.2 of [4]. The NTK predictors are also universal and are a subset of the random feature models treated in [5], which derives the neural scaling laws for these models. Extrapolating these laws out as per [6] for specific tasks shows that the "floor" is always below human error rates, but this is still empirical because it works with the ill-defined definition of superintelligence that is "better than humans in all contexts". [1] Hutter, M. (2005). Universal artificial intelligence: Sequential decisions based on algorithmic probability. Springer Science & Business Media. [2] https://arxiv.org/abs/1912.10077 [3] https://openreview.net/pdf?id=Vib3KtwoWs [4] https://arxiv.org/abs/2006.14548 [5] https://arxiv.org/abs/2210.16859 [6] https://arxiv.org/abs/2001.08361
Original comment The Problem with LLMs
YetAnotherNick
Neutral mention · rule
It wasn't a lie, it was a misrepresentation of the total cost. It's not hard to calculate the cost of the training though. It takes 6 * active parameters * tokens flops[1]. To get number of seconds you can divide by Flops/s * MFU, where MFU is around 45% for H100 for large enough models[2]. [1]: https://arxiv.org/abs/2001.08361 [2]: https://github.com/facebookresearch/lingua
Original comment GPU-rich labs have won: What's left for the rest of us is distillation
Ukv
Neutral mention · rule
> The new model seems to prove that longstanding rumors of diminishing returns in training unsupervised-learning LLMs were correct and that the so-called "scaling laws" cited by many for years have possibly met their natural end. The commonly-cited scaling laws[0] predict diminishing returns from scale. Fairly uncontroversial for instance that going from 1 GPU to 2 GPUs gives a larger improvement than going from 101 GPUs to 102 GPUs. Same with, say, computer graphics. I feel scaling laws are getting conflated with the theory of an exponential "hard takeoff" of self-improvement (which isn't particuarly well-founded in my opinion). [0]: https://arxiv.org/pdf/2001.08361
Original comment "It's a lemon"–OpenAI's largest AI model ever arrives to mixed reviews
Workaccount2
Neutral mention · rule
>if your methodology is flawed you aren't gonna get any more significant returns. The problem with this statement is that predictions made about scaling 5 years ago have held true[1]. We keep adding parameters, adding compute, and the models keep getting more capable. The flaws of LLM's from 2024 are not what is relevant. Just like the flaws of LLMs from 2021 were not relevant. What is relevant is the rate of change, and the lack of evidence that things won't continue on this steep incline. Especially if you consider that GPT4 was sort of a preview model that motivated big money to make ungodly investments to see how far we can push this. Those models will start to show up over the next 2 years. If they break the trend and the scaling flops, then I think a lot of air is gonna blow out of the bubble. [1]https://arxiv.org/pdf/2001.08361
Original comment Understanding the Limitations of Mathematical Reasoning in LLMs
logicchains
Neutral mention · rule
>I think the data will show that it is going to take logarithmic costs in either train-time or inference-time for linear or even sub-linear improvements. The data already shows this, it's a well-established result: https://arxiv.org/abs/2001.08361 . Well, less established for inference-time compute, but OpenAI even said they saw the same thing for inference compute in their release post for o1. But there's still room for at least an order of magnitude speedup with specialised hardware (as opposed to more general-purpose GPUs) and ternary nets.
Original comment OpenAI in throes of executive exodus as three walk at once
espadrine
Neutral mention · rule
> New LLMs don't even rely that much on that aforementioned older architecture Don’t they all indicate being based on the transformer architecture? > not entirely because of transformers but because of the hardware Kaplan et al. 2020[0] (figure 7, §3.2.1) shows that LSTMs, the leading language architecture prior to transformers, scaled worse because they plateau’ed quickly with larger context. [0]: https://arxiv.org/abs/2001.08361
Original comment OpenAI to become for-profit company
findingMeaning
Neutral mention · rule
> Why? I’m utterly baffled by this. Yes, me too! I am surprised people are unaware of the empirical power-scale law. [1] Currently, Anthropic's new model has proven it by being better than GPT-4 on some tasks. Yet, the old people are saying it is okay. They are saying engineering is not just coding. What they don't seem to grasp is there won't be a necessity for as many engineers. If people are not willing to accept juniors, where will the next generation of seniors come from? No clue about that. Probably, that spot is reserved for the brilliant minds graduating from R1 institutes. What about the average Joe? Not everyone has the fortune to work on very specialized domain…
Read the full comment
skills. People just don't get it easily. Everyone in academia has admitted to productivity boost by 30-40%. And here we are talking about hard research. Imagine, what it looks for regular job which is managing and maintaining codebases. The severity of this situation is truly alarming not for the greybeards but for us, Zoomers. With climate crisis, AI, geopolitical instability, it feels really great to be alive. [1]: https://arxiv.org/abs/2001.08361
Original comment Ask HN: How worried are you about being replaced by an LLM
hiddencost
Neutral mention · rule
Number go up. More seriously, it looks like a potential reduction in the cost to train a neural network. You can think of every meaningful step forward in deep learning as a reduction in the cost of training, or an improvement in the ability of the signal to propagate. https://arxiv.org/abs/2001.08361
Original comment Matryoshka Representation Learning
duchenne
Neutral mention · rule
The most important paper to understand this issue is "Sacling Laws of Neural Language Models" by Open AI in 2020 [1]. Many consider it the most important paper that predicted the high performance of modern LLMs. This paper shows how the loss decreases when you increase the model size, compute, or training dataset size. From the article: > Convergence is inefficient: When working within a fixed compute budget C but without any other restrictions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence. It clearly states that when you are limited by your training time compute, you should under-train your model. [1] https://arxiv.org/abs/2001.08361
Original comment Do large language models need all those layers?
circuit10
Neutral mention · rule
If you want theory there’s this: https://arxiv.org/abs/2001.08361 (I haven’t actually read this but I know roughly what it’s about) It’s saying that so far the abilities of an LLMs have scaled up with its parameter count and training data size. Of course there’s no way to be sure without actually training larger models but I don’t see why the point where it stops would be just after our current best LLMs. Many properties have already emerged from making it bigger so I don’t see why this would be the exception
Original comment How Is LLaMa.cpp Possible?
ppaattrriicckk
Neutral mention · rule
I love bashing Intel as much as the other person, but looking at the number of tokens LLaMA[1] was trained on (1,400B tokens), the number of parameters of GPT-3[2] vs. the number of tokens it was trained on (175B params vs. 300B), it's not like the announced 1,000B param model is unreasonable. Taking 175/300*1400 yields 816B parameters - which is fairly close to 1,000B. Not outright the most efficient utilization of data, but as others have mentioned, there might still be something left to gain (from not solely optimizing for compute). E.g. look at Figure 9 in [3]. Although the model trained on the largest model obviously utilized the data most efficiently, it's not…
Read the full comment
perfectly clear - to me at least - whether some over-parameterization will necessarily lead to a decrease of test/out-of-sample performance. Of course, LLaMA was trained on way fewer parameters relative to data set size. I've just mentioned LLaMA as a point of reference to the largest dataset known to me. [1] - https://arxiv.org/pdf/2302.13971.pdf [2] - https://arxiv.org/pdf/2005.14165.pdf [3] - https://arxiv.org/pdf/2001.08361.pdf EDIT: Formatting of references
Original comment Intel Announces Aurora GenAI, ChatGPT Competitor with 1T Parameters
sandGorgon
Neutral mention · rule
hey thanks for your reply. umm...so does OpenAI. In fact this is OpenAI discovery from [1]: >Convergence is inefficient: When working within a fixed compute budget C but without any other restric- tions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3). Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as D ∼ C0.27 with training compute. (Section 6) >We have also tested our models on a set of additional text data…
Read the full comment
distributions. The test loss on these datasets as a function of model size is shown in Figure 8; in all cases the models were trained only on the WebText2 dataset. We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2. We find that generalization depends almost exclusively on the in-distribution validation loss, and does not depend on the duration of training or proximity to convergence. We also observe no dependence on model depth (see Appendix D.8) P.S. Not trolling. genuinely trying to learn. [1] https://arxiv.org/abs/2001.08361
Original comment LLaMA: A foundational, 65B-parameter large language model
hackinthebochs
Neutral mention · rule
>If you took GPT3's architecture and scaled it down to the size and training set of a typical auto-predict, it would produce near identical results. This is almost certainly not true. The number of parameters is an important feature related to the quality of output. If you scaled the architecture down significantly, it would be significantly less capable[1]. But perhaps I misunderstand your point. >Likewise, if we took an auto-predict architecture from 8 years ago, scaled it up to the size of GPT3 and could train it on GPT3's training set, it would produce similar output to GPT3 This is also not true. The transformer is a key piece in the emergent abilities of language…
Read the full comment
models. The difficulties in scaling RNNs are well known. Self-supervised learning is powerful, but it needs to be paired with a flexible architecture to see the kinds of gains we see with LLMs. Stacked Transformers with self-attention are extremely flexible in finding novel circuits in service to modelling the training data. The question is how to characterize this model in a way that doesn't short-sell what it is doing. Reductively describing it in terms of its training regime is just to treat the resulting model as explanatorily irrelevant. But the complexity and the capabilities are in the information dynamics encoded in the model parameters. The goal is to understand that. [1] https://arxiv.org/abs/2001.08361
Original comment Bing AI Can't Be Trusted
airgapstopgap
Neutral mention · rule
They talk specifically of scaling laws investigated in papers like OpenAI's "Scaling Laws for Neural Language Models" (2020)[1] and Deepmind's "An empirical analysis of compute-optimal large language model training" (2022)[2] which showed that, as far as we could see at the time, for any sensibly designed language model nuances of architecture do not matter nearly as much as the sheer scale of the model, amount of compute spent on training it, and volume of training dataset (or, indeed, that they do not matter at all, or even that attempts to add some cleverness via architecture ultimately handicap the model at production scale). These papers have introduced so-called…
Read the full comment
"scaling laws" – predictable, robust relationships between scaling parameters and loss (which is meaningfully related to textual coherence and apparent "understanding" in zero-shot tasks), and for many have heralded the age of throwing compute at the problem without academic tinkering, and possibly the final stretch before human-level text processing. See also Rich Sutton's Bitter Lesson [3]. In this sense it is remarkable when the curve predicted from "scaling laws" gets broken through, transcended, since it suggests there still exists a better Pareto frontier for text prediction. I see how this can appear normal; but the thing is, scaling laws have been well-validated. We are no longer in the early era to expect to reap substantial returns from something as simple as changing training objective. 1. https://arxiv.org/abs/2001.08361 2. https://www.deepmind.com/publications/an-empirical-analysis-... 3. http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Original comment Transcending Scaling Laws with 0.1% Extra Compute

Keep following the thread.

More in Language models

More from the same topics.