Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo
João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov
2022 · arXiv
At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We start by providing a simple weight construction that shows the equivalence of data transformations induced by 1) a single linear self-attention layer and by 2) gradient-descent (GD) on a regression loss. Motivated by that construction, we show empirically that when training self-attention-only Transformers on simple regression tasks either the models learned by GD and Transformers show great similarity or, remarkably, the weights found by optimization match the construction. Thus we show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass. This allows us, at least in the domain of regression problems, to mechanistically understand the inner workings of in-context learning in optimized Transformers. Building on this insight, we furthermore identify how Transformers surpass the performance of plain gradient descent by learning an iterative curvature correction and learn linear models on deep data representations to solve non-linear regression tasks. Finally, we discuss intriguing parallels to a mechanism identified to be crucial for in-context learning termed induction-head (Olsson et al., 2022) and show how it could be understood as a specific case of in-context learning by gradient descent learning within Transformers. Code to reproduce the experiments can be found at https://github.com/google-research/self-organising-systems/tree/master/transformers_learn_icl_by_gd .
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
Would that make in-context learning a superset or a subset of pretraining? This paper claimed transformers learn a gradient-descent mesa-optimizer as part of in-context learning, while being guided by the pretraining objective, and as the parent mentioned, any general reasoner can bootstrap a world model from first principles. [0] https://arxiv.org/pdf/2212.07677
I used to agree with you. The paper that made me unsure was "Transformers learn in-context by gradient descent" [1]. Basically, the model learns weights that let it run gradient descent at inference time in order to do in context learning. If transformers can learn this, then I think they can learn almost anything given enough compute/parameters/data. Of course, even if this is true then it's possible that there simply isn't enough high quality data, or that the amount of compute required is beyond our current hardware. [1] https://arxiv.org/abs/2212.07677
I agree, the in-context, few shot learning is the most interesting thing a about these transformer models, they have basically learned to learn (meta-learning). Some additional interesting papers about it: Transformers learn in-context by gradient descent: https://arxiv.org/abs/2212.07677 What Can Transformers Learn In-Context? A Case Study of Simple Function Classes: https://arxiv.org/abs/2208.01066 "We also show that we can train Transformers to in-context learn more complex function classes -- namely sparse linear functions, two-layer neural networks, and decision trees -- with performance that matches or exceeds task-specific learning algorithms."
The article doesn't understand large language models. It makes an assumption about how they work and uses that assumption to draw a conclusion. Most likely, as found in small models [0], large autoregressive transformers understand machine learning algorithms well enough to apply them to their inputs to achieve in-context learning (few-shot prompting) by performing gradient descent within the overall network [1]. This is fundamentally different from static predictors; LLMs are dynamic modelers that learn from context in order to better predict tokens. [0] https://arxiv.org/pdf/2211.15661 [1] https://arxiv.org/abs/2212.07677
> Why do people say they don't now how it works "Curve Fitting" is the objective, the function encoded in the weights is the solution, and not actually well understood. See work from Anthropic[1] and Google[2] that explores this. As an analogy consider applying the same argument to the AlphaGo value function. It's "just" fitting a bunch of curves to the statistics of millions of self-played games. However, to effectively capture those statistics the network needed to develop a bunch of heuristics. Needless to say these heuristics are not understood (else we'd already know the principles needed to play at AlphaGo's level), and are not just exhaustive lists of statistical…
trends but more like strategies[3]. Recent work[4] strongly suggests that "grokking" (a striking but not unnatural[5] form of generalization) involves networks transitioning from memorized statistics/solutions to a general solution. The curve fitting perspective would totally miss all this for a comfortable but misleading story: "the objective is curve fitting so it's just interpolating data points". [1] https://transformer-circuits.pub [2] https://arxiv.org/abs/2212.07677 [3] https://www.pnas.org/doi/10.1073/pnas.2206625119 [4] https://arxiv.org/abs/2301.05217 [5] https://arxiv.org/abs/2210.01117
Clearly it doesn't need to have seen the exact input before because you can ask GPT-4 to add together two 6 digit numbers and it will get it right. More impressively, if you generate a random small neural network and then put some samples in the prompt it can do a surprisingly good job of predicting the result of putting other inputs into the network. Presumably doing something that looks a bit like gradient decent, see https://arxiv.org/pdf/2212.07677.pdf Solely in order to predict the next token, LLMs are learning an incredibly sophisticed model of both computation and the outside world.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.