Skip to content
The methodOpen data
Back to the collection

Locating and Editing Factual Associations in GPT

Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov

2022 · arXiv

Language modelsStatistics & probabilityRobotics
Read the paper
HN score
1.5
Mentions
6
Readers
6
Months
6
What’s in a score?

From the abstract

We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at https://rome.baulab.info/

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

killerstorm
Strong recommendation · rule
There's definitely scientific insight and analysis. E.g. "In-context Learning and Induction Heads" is an excellent paper. Another paper ("ROME") https://arxiv.org/abs/2202.05262 formulates hypothesis over how these models store information, and provide experimental evidence. The thing is, a 3-layer MLP is basically an associative memory + a bit of compute. People understand that if you stack enough of them you can compute or memorize pretty much anything. Attention provides information routing. Again, that is pretty well-understood. The rest is basically finding an optimal trade-off. These trade-off are based on insights based on experimental data. So this architecture is not so much accidental as it is general. Specific representations used by MLPs are poorly understood, but there's definitely a progress on understanding them from first principles by building specialized models.
Original comment Llama3 implemented from scratch
stalfie
Neutral mention · rule
Sure, here are some citations: https://arxiv.org/abs/2401.03910?utm_source=chatgpt.com https://www.frontiersin.org/journals/psychology/articles/10.... https://arxiv.org/pdf/2408.04666 https://arxiv.org/abs/2402.00901 https://ar5iv.labs.arxiv.org/html/2407.11015 https://ar5iv.labs.arxiv.org/html/2202.05262 https://www.sciencedirect.com/science/article/abs/pii/S13646... My point was not to argue one point or another about LLM intelligence, but to push against the notion that you have to be "delusional" to even argue that it is possible that LLMs can qualify as intelligent (although the op prefaced it with "actually"). That's mainly what irked me about the original comment,…
Read the full comment
the arrogance of dismissing everyone even having the discussion as insane, as if there's no legitimate argument to be made. And this was mainly the point I was arguing. However since we're on the topic, I also happen to think it's intellectually lazy to dismiss the Anthropic interpretability teams work as "delusional" simply because they have a conflict of interest. Of course, that is not an irrelevant fact, but much of their original work has since been replicated by independent entites (eg. https://arxiv.org/abs/2510.01246). Until they publish something that turns out to be fraudulent, I think it's reasonable to consider Anthropic's paid people a very relevant, and in fact excellent part of interpretability discourse. Dismissing all opinions where there is a perceived conflict of interest is a pleasant cognitive bias to have, but reality is often more nuanced than that.
Original comment AI 2040: Plan A
filterfiber
Neutral mention · rule
The most likely "real solution" is going to be using various tricks and finetuning on higher context lengths to just extend the context window. However I will say knowledge graphs are becoming more popular. I made a comment the other day with a list of some of the popular methods: https://news.ycombinator.com/item?id=38476596 I completely forgot to mention ROME in my last comment, where you can modify facts within an LLM https://arxiv.org/pdf/2202.05262.pdf Note that there could be something more recent then these that I missed, but as far as I know knowledge-graphs/RAG are what most people are currently using, but there's a lot of work being focused on extending the context window.
Original comment What Is Retrieval-Augmented Generation a.k.a. RAG?
numeri
Neutral mention · rule
This post isn't a scientific investigation. It's someone playing around with a black-box model with few to little controls – which is unfortunately the only thing we can do when experimenting with GPT-4, which does excuse this partially. Unfortunately, though, the author seems unaware of the actual state of research on the actual mechanics of how LLMs store knowledge and specifically binary relations. The ROME paper[1], among others, shows that the feed-forward layers function as a key-value store, where the feed-forward's up projection of the last token in a noun phrase (say, "the Eiffel Tower") acts as a key, which when multiplied by the down projection, produces a…
Read the full comment
value that contains information the model knows about the subject, which is then added into the residual stream/hidden representation. A paper building on that work[2] then went on to show that it's usually the self-attention layers that use the relational phrase (say, "is in") to extract the relevant knowledge from the feed-forward layer's output (in this example, hopefully "Paris"). This mechanistic understanding makes it really obvious why the reversal curse occurs – using matrix multiplication as a key-value store requires having a fully separate key-value pair to look up the reversed relation. [1] https://arxiv.org/abs/2202.05262 [2] https://arxiv.org/abs/2304.14767v1
Original comment Is the reversal curse in LLMs real?
KhoomeiK
Neutral mention · rule
The title made me think this was an approach that used memory editing techniques (e.g. ROME [1]) to allow an LLM's neural memory (not just its context) to change over the course of conversation. Pretty happy to realize that this is just a fancy RAG work—will be building my version of MemEditGPT soon. [1] https://arxiv.org/abs/2202.05262
Original comment MemGPT – LLMs with self-editing memory for unbounded context
nl
Neutral mention · rule
> does anyone know of ML directions that could add any kind of factual confidence level to ChatGPT and similar? Yes. It's a very active area of research. For example: Discovering Latent Knowledge in Language Models Without Supervision (https://arxiv.org/abs/2212.03827) shows an unsupervised approach for probing a LLM to discover things it thinks are facts Locating and Editing Factual Associations in GPT (https://arxiv.org/pdf/2202.05262.pdf) shows an approach to editing a LLM to edit facts. Language Models as Knowledge Bases? (https://aclanthology.org/D19-1250.pdf) is some slightly older work exploring how well LLMs store factual information itself.
Original comment ChatGPT is falsely recommending us for a service we don't provide

Keep following the thread.

More in Language models

More from the same topics.