Skip to content
The methodOpen data
Back to the collection

Progress measures for grokking via mechanistic interpretability

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt

2023 · arXiv

PsychologyAlgorithmsMachine learning
Read the paper
HN score
0
Mentions
10
Readers
7
Months
7
What’s in a score?

From the abstract

Neural networks often exhibit emergent behavior, where qualitatively new capabilities arise from scaling up the amount of parameters, training data, or training steps. One approach to understanding emergence is to find continuous \textit{progress measures} that underlie the seemingly discontinuous qualitative changes. We argue that progress measures can be found via mechanistic interpretability: reverse-engineering learned behaviors into their individual components. As a case study, we investigate the recently-discovered phenomenon of ``grokking'' exhibited by small transformers trained on modular addition tasks. We fully reverse engineer the algorithm learned by these networks, which uses discrete Fourier transforms and trigonometric identities to convert addition to rotation about a circle. We confirm the algorithm by analyzing the activations and weights and by performing ablations in Fourier space. Based on this understanding, we define progress measures that allow us to study the dynamics of training and split training into three continuous phases: memorization, circuit formation, and cleanup. Our results show that grokking, rather than being a sudden shift, arises from the gradual amplification of structured mechanisms encoded in the weights, followed by the later removal of memorizing components.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

10 selected excerpts · 10 total mentions

soulofmischief
Neutral mention · rule
This is not a correct approximation of what happens inside an LLM. They form probabilistic logical circuits which approximate the world they have learned through training. They are not simply recalling stored facts. They are exploiting organically-produced circuitry, walking a manifold, which leads to the ability to predict the next state in a staggering variety of contexts. As an example: https://arxiv.org/abs/2301.05217 It's not hard to imagine that a sufficiently developed manifold could theoretically allow LLMs to interpolate or even extrapolate information that was missing from the training data, but is logically or experimentally valid.
Original comment Claude is not a senior engineer (yet)
RoboTeddy
Neutral mention · rule
> But the ways in which a LLM can “talk about” the universe (and everything it contains) are limited to the ways in which humans have previously talked about the universe. This is often said, but it isn't so. The task of predicting the next token in human speech really well requires immense intelligence — potentially far more intelligence than possessed by the original speaker! Imagine yourself engaging in the task of listening to someone who isn't that smart speak and then trying to figure out what they'll say next — in doing so, you might make all sorts of extrapolations about the person, their motivations, their manner, their dialect, etc — calling on all sorts of…
Read the full comment
internal models that you've built up about people over time. This is what models are being trained to do when we train them on predicting tokens. There are concrete examples of models inventing new ways of thinking that are not described in their training set. For example, when training a transformer from scratch to perform addition mod P (and having no training data other than examples of addition mod P), the transformer was able to discover the use of discrete fourier transforms and trigonometric identities [1]. As we can see, neural nets can build all sorts of internal mental models that no one explained to them beforehand. These internal mental models can then be elicited and used for other purposes by e.g. fine-tuning. I think a good mental model for transformers/neural nets is that they're automatic scientists. They figure out ways of modeling things in order to predict the output from the input — which is what scientists do! As part of this, they can de-facto discover new theories, and come to rely on the theories that prove useful in their prediction task. Also, not all tokens in the training set are from human speech, so models are being trained to model all manner of data-generating processes. [1] https://arxiv.org/pdf/2301.05217.pdf
Original comment The Myth of AI Omniscience
YeGoblynQueenne
Neutral mention · rule
Arxiv preprint: https://arxiv.org/pdf/2301.05217.pdf OpenReview.net review: https://openreview.net/forum?id=9XFSbDPmdW I'm skimming the paper and the OpenReview reviews [1]. The grokking claim is, as always, easily dismissed: it obtains on a small, artificial dataset when training a small, contrived model. There's no reason to get excited until results generalise to larger models and real-world architectures, trained on real-world datasets. Unfortunately this is a lot to ask because the "mechanistic interpretability" approach doesn't seem to have the power to analyse such mega-structures. So I don't think anyone has learned anything about "grokking" from the paper: those…
Read the full comment
who want to believe, will still believe, those who want evidence, will not find evidence. I'm more interested in the claims of algorithm learning. There's an obvious test that the authros haven't tried: hand-code the algorithm they reverse-engineered and compare it to the trained model. To clarify, the authors claim that they reverse engineered a certain algorithm to perform modular arithmetic off the weights of their trained neural net that was showing 100% performance on their training set. The algorithm, which they call the "Fourrier multiplication algorithm" is outlined in figure 1 and further elucidated in Section 3.1. The algorithm goes over some frequencies, five of which seem to be predictive of most of the model's performance. So both the algorithm and its parameters are known (or anyway assumed). In that case, it should be possible to hand-code the algorithm and run it [2]. Then, given some set of inputs X, taken from the experiments' test set, first a) test the algorithm's accuracy and then b) compare the algorithm's output to the outputs of the model, given the same inputs. The point of this is to check, respectively, that a) the proposed algorithm is a correct addition algorithm (and that it is an algorithm, i.e. it always terminates) and b) that its behaviour matches exactly the behaviour of the model it is claimed to be encoded in. Note that the authors base their certainty on the correctness of the algorithm, and its fidelity as an algorithmic encoding of their model, to a series of ablation studies. It's genuinely refreshing to observe this almost extinct species in the wild, but it is not enough. Not when there is a stronger test available (the one I point out above). I'm almost disappointed to see an ablation study, because it looks like an excuse to not go the whole hog. In the same spirit, what is entirely missing is a proof of correctness of the algorithm. I have never seen this algorithm before and eyballing it, it "looks OK", but that's not how algorithmic correctness is normally ascertained. There's all this maths in the paper, but not a proof in sight. What's the point of maths then, to calculate large numbers? _________________________________ [1] The paper was accepted for publication in ICLR 2023 so congratulations to the authors. [2] To be perfectly clear about what I mean: I mean it should be possible to code the algorithm as a computer program in a language like, e.g., Python. The "algorithm" is an encoding of the trained model in a different, but equivalent form. It's now possible to encode it in yet another different, but equivalent, form, and run it, to perform "inference".
Original comment Tiny Transformer trained for addition learns bizarre addition algorithm
sanxiyn
Neutral mention · rule
If you are interested in grokking you should have a look at this: Progress measures for grokking via mechanistic interpretability. https://arxiv.org/abs/2301.05217
Original comment Grokking in Julia
eiz
Neutral mention · rule
> Where is the connection between computational details and the model's high-level behavior? Do we even know? This is an active area of study ("mechanistic interpretability") and it's very early days. For instance here's a paper I read recently that tries to explain how a very simple transformer learns how to do modular arithmetic: https://arxiv.org/abs/2301.05217 Curious what interesting results people are aware of in this area.
Original comment What Are Transformer Models and How Do They Work?
cmorez
Neutral mention · rule
> Why do people say they don't now how it works "Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this. As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles needed to play at AlphaGo's level), and are not just exhaustive lists of statistical…
Read the full comment
trends but more like strategies[3]. Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution. The curve fitting perspective would totally miss all this for a comfortable but misleading story: "the objective is curve fitting so it's just interpolating data points". [1] https://transformer-circuits.pub [2] https://arxiv.org/abs/2212.07677 [3] https://www.pnas.org/doi/10.1073/pnas.2206625119 [4] https://arxiv.org/abs/2301.05217 [5] https://arxiv.org/abs/2210.01117
Original comment Do Large Language Models learn world models or just surface statistics?
soulofmischief
Neutral mention · rule
Nanda, et al. successfully recovered the exact mechanism through which a transformer learned to carry out modular addition. [0] Transformers are all about the training data, and we will increasingly learn that structuring the order in which data is learned matters a lot. But it's clear that transformers are absolutely capable of encoding generalized solutions to arithmetic. Given the right tokenization scheme and training regimen, we can absolutely create LLMs which have statistically sound arithmetic capabilities. I still wouldn't trust a stochastic model over the algorithmic certainty of a calculator, but what's more important for mathematicians is that these models can reason about complex problems and help them break new ground on hard mathematical problems by leveraging the full statistical power of their weights. [0] https://arxiv.org/abs/2301.05217
Original comment Understanding the Limitations of Mathematical Reasoning in LLMs
sanxiyn
Neutral mention · rule
We are building debuggers (really more like disassemblers) and we are reverse engineering LLMs. There is no fundamental difficulty, but it takes time. For example, one layer transformer eventually learns to do modular addition completely correctly. Looking at weights, it does so by doing rotation on unit circle. For a + b = c mod p, probability of c is proportional to cos((a+b-c) 2π/p), which works because cosine is maximized when the argument is multiple of 2π. As you can see, there is no fundamental difficulty, but it is also not trivial. https://arxiv.org/abs/2301.05217
Original comment We need a much more sophisticated debate about AI

Keep following the thread.

More in Psychology

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score