Damai Dai, Yutao Sun, Li Dong
Yaru Hao, Shuming Ma, Zhifang Sui, Furu Wei
2022 · arXiv
Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input without parameter updates. Despite the great success in performance, its working mechanism still remains an open question. In this paper, we explain language models as meta-optimizers and understand in-context learning as implicit finetuning. Theoretically, we figure out that Transformer attention has a dual form of gradient descent. On top of it, we understand ICL as follows: GPT first produces meta-gradients according to the demonstration examples, and then these meta-gradients are applied to the original GPT to build an ICL model. We comprehensively compare the behaviors of in-context learning and explicit finetuning on real tasks to provide empirical evidence that supports our understanding. Experimental results show that in-context learning behaves similarly to explicit finetuning from multiple perspectives. Inspired by the dual form between Transformer attention and gradient descent, we design a momentum-based attention by analogy with gradient descent with momentum. The improved performance over vanilla attention further supports our understanding from another perspective, and more importantly, shows the potential to utilize our understanding for future model design. The code is available at \url{https://aka.ms/icl}.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
20 selected excerpts · 21 total mentions
That's false. It has been shown that LLMs can perform e.g. gradient descent internally [1], which can explain why they are so good at few shot prompting. The universal approximation theorem already tells us that a single layer is sufficient to approximate any function, so it should come as no surprise that modern deep networks with many layers should be able to perform iterative optimisations. [1] https://arxiv.org/abs/2212.10559
>but they can only reason using their memories and the prompt. Eh no. https://arxiv.org/abs/2212.10559 >But if you try very hard you can find "held out" data and when you test on it, GPT4 stops looking so smart: This can be done to anybody. This can be done to you. It's not a gotcha. Nobody is saying GPTs don't/can't memorize.
> The “learning” in “In context learning” is not the same concept as learning during the network “training”. The former is loosely named and does not have any impact on what’s actually “learned” by the model. Actually (and very surprisingly!) it is related. See the "Similarity of Attention Output Updates" and "Similarity of Attention Map" metrics in https://arxiv.org/abs/2212.10559 (linked above) where they show that in-context learning actually changes the attention map in a similar way to traditional fine tuning(!!). Obviously it can't modify the weights of the model on disk, but after ICL new passes through the model are treated quite similarly to how they would be if it has been fine tuned (again: this is astonishing!)
No there is reasoning in LLMs. This is actually proven through several scientific papers. I can't say whether it's using reasoning for specific answers. I'm sure many times it just regurgitates something that is known... But saying that LLMs in general can and do reason is categorically true. LLMs do attempt to extract actual mental models of the subject of the text. See here: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559 For your specific case, given the conditions, another explanation that fits as well is that the LLM simply needs more data to build a more complete model for elixir. Once it does that I'm sure it can respond with completely novel elixir code as it already does with python.
>> LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer If his arguments are entirely based on this, then it's not fully correct: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
Bingo. There’s a recent paper that posits language models are meta learners where the transformer layer is approximating stochastic gradient descent updates from the inputs—this is what allows them to perform in-context learning. [1] If that’s true, then it is going to be impossible to prevent a sufficiently large language model from being prompt-hacked. You just need to find a collection of input tokens that moves the network into the region of undesirable behavior you want to promote. This is mathematically equivalent to retraining the network to misbehave. Prompt-hacking is analogous to an AI virus—it exploits the fundamental mechanism of operation in the transformer-based language model as a vulnerability. Worse, if this paper is true then this is an intrinsic property of the mathematics of a transformer layer—in which case this kind of vulnerability can never be eliminated. [1] https://arxiv.org/abs/2212.10559
- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
I saw this paper’s abstract and was reminded of your comment: https://arxiv.org/abs/2212.10559 > Experimentally, we comprehensively compare the behavior of ICL and explicit finetuning based on real tasks to provide empirical evidence that supports our understanding
The answer to this is: "we don't really know as its a very complex function automatically discovered by means of slow gradient descent, and we're still finding out" Here are some of the fun things we've found out so far: - GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
1. I don't understand how LLMs work. 2. I don't understand how ChatGPT works, but I have used it a few times. 3. I will use ChatGPT as the absolute measure of what LLMs are capable of. --- 1. I don't understand intelligence. 2. Humans are intelligent, humans can learn to do math. 3. LLMs are not good at math. 4. LLMs are not intelligent, they're just text compression. --- 1. I don't understand how LLMs work. 2. I have a decent grasp of how image compression works. 3. I will use my grasp of image compression to pretend LLMs are text compression. 4. I will apply all limitations of image compression to LLMs. 5. "What use is there in having something that rephrases the Web?…
If we were losing our access to the Internet forever and had to store a copy on a private server with limited space, a large-language model like ChatGPT might be a good solution, assuming that it could be kept from fabricating. But we aren’t losing our access to the Internet. So just how much use is a blurry jpeg, when you still have the original?" --- What's funny is that the author has produced exactly what many claim LLMs to be useless for - flowery words that seem true but are not. I don't think this should've been published. These are both good reads if you find yourself tending to agree with the author - - Emergent Abilities of Large Language Models - https://arxiv.org/abs/2206.07682 - Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers - https://arxiv.org/abs/2212.10559v2
With transformer-based model, their inner-state is a deterministic function (the features encoded by the Neural Networks weights) applied to the text-generated up-until the current-time step, so it's relatively easy to know what they currently have in mind. For example if the neural network has been generating sad music, its current context which is computed from what it has already generated will light-up the the features that correspond to "sad music". And in turn the fact that the features had been lit-up will make it more likely to generate a minor chord. The dimension of this inner-state is growing at each time-step. And it's quite hard to predict where it will go.…
For example if you prompt it (or if it prompts itself) "happy music now", the network will switch to generating happy music even if in its current context there is still plenty of "sad music" because after the instruction it will choose to focus only on the recent more merrier music. Up until recently, I was quite convinced that using a neural network in evaluation mode (aka post training with its weight frozen) was "(morally) safe", but the ability of neural network of performing few-shot learning changed my mind (The Microsoft paper in question : https://arxiv.org/pdf/2212.10559.pdf : "Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers" ). The idea in this technical paper is that with attention mechanism even in forward computation there is an inner state that is updated following a meta-gradient (aka it's not so different from training). Pushing the reasoning to the extreme would mean that "prompt engineering is all you need" and that even with frozen weight with a long enough time-horizon and correct initial prompt you can bootstrap a consciousness process. Does "it" feels something ? Probably not yet. But the sequential filtering process that Large Language Models do is damn similar to what I would call a "stream of consciousness". Currently it's more like a markov chain of ideas flowing from idea to the next idea in a natural direction. It's just that the flow of ideas has not yet decided to called itself it yet.
For those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the context with examples. Traditionally this "context window" has been a few thousand tokens, and GPT-4 recently extended that to 32,000 tokens. That is useful, but if you wanted to say load all of a companies documents and ask questions it doesn't really work…
because this overflows the context. But at 2M tokens there's a whole range of applications that become possible. [1] Language Models are Few-Shot Learners: https://arxiv.org/abs/2005.14165 [2] Language Models Secretly Perform Gradient Descent as Meta-Optimizers: https://arxiv.org/abs/2212.10559
I think there is an issue in vocabulary size though. If I'm correct in thinking that the OP is proposing that what Llama does ("Notably, we split all numbers into individual digits") explicitly this allows the model to treat a number as a sequence of digits so it can learn how math works. This is especially important with very rare numbers. Take a number the GPT has never (or hardly ever) seen in its training data: 832408345770928764 The GPT-3 tokenizer tokenizes[1] that into: 83,24,08,345,7,709,287,64 To some degree this is forced to occur by the use of raw BPE encoding and the vocabulary size (175K in the case of GPT-3). Now consider the string: 832408345770928764 +…
37 The model presumably has learnt something like "if all tokens are in this area (where the area is "numbers" in the token-space) and they are followed by a + sign then we don't just append the string, instead we swap the last token for another one" But of course this is insufficient in this case - it needs to learn carrying rules to also increment the next token. As is speculated in https://arxiv.org/abs/2212.10559, it's possible there's a relationship between the depth of the model and the length of chained rules it can learn, and because of the number of multi-digit tokens it has learnt these rules are unnecessarily complex, and incomplete. If these tokens were instead single digits the rules would be much simpler, and it's possible the model could actually learn the real rules of math instead of the subset of semi-memorized things it has at the moment. [1] https://platform.openai.com/tokenizer
I don't think that's clear at all. https://arxiv.org/abs/2212.10559 shows a LLM is doing gradient descent on the context window at inference time. If it's learning relationships between concepts at runtime based on information in the context window then it seems about as useful to say it is a Markov chain as it is to say that a human is a Markov chain. Perhaps we are, but the "current state" is unmeasurably complex.
> We know how LLMs work and they auto-complete text. This isn't true. See eg[1] It seems to be a common misconception that the training objective "predict the next word" implies something about the capabilities of a LLM. As [1[] shows, instead they perform very sophisticated calculations on their compressed knowledge representation to try to generate acceptable text. These capabilities can be accessed for other purposes too. [1] "Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers" https://arxiv.org/abs/2212.10559
In context learning is already implicit finetuning. https://arxiv.org/abs/2212.10559. It's very questionable to what extent continuous training is necessary past a threshold of intelligence.
- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382 - GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559
> I must have missed the part when it started doing anything algorithmically. Yeah. "Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers" https://arxiv.org/abs/2212.10559 @dang there's something weird about this URL in HN. It has 35 points but no discussion (I guess because the original submission is too old and never got any traction or something)
I didn't agree with the OP but just noting this is wrong: > These nets regurgitate the most average possible output given inputs. By definition. You seem to be confusing the training objective with the capabilities. For example: > Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers https://arxiv.org/abs/2212.10559
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.