Skip to content
The methodOpen data
Back to the collection

Tracr: Compiled Transformers as a Laboratory for Interpretability

David Lindner, János Kramár, Sebastian Farquhar

Show all 6 authors

Matthew Rahtz, Thomas McGrath, Vladimir Mikulik

2023 · arXiv

Language modelsEconomicsAlgorithms
Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

We show how to "compile" human-readable programs into standard decoder-only transformer models. Our compiler, Tracr, generates models with known structure. This structure can be used to design experiments. For example, we use it to study "superposition" in transformers that execute multi-step algorithms. Additionally, the known structure of Tracr-compiled models can serve as ground-truth for evaluating interpretability methods. Commonly, because the "programs" learned by transformers are unknown it is unclear whether an interpretation succeeded. We demonstrate our approach by implementing and examining programs including computing token frequencies, sorting, and parenthesis checking. We provide an open-source implementation of Tracr at https://github.com/google-deepmind/tracr.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

nojvek
Neutral mention · rule
There are papers written about this. https://openreview.net/pdf/0c9a0f445d430a4d2bf9a08078c435091... Without a loop - an LLM takes a fixed set of tokens and produces a fixed set of tokens. It has some internal state. So without a loop an LLM cannot be a general purpose computer (Turing machine). Turing machines are simple. They need 3 things - infinitely long tape, ability to have internal state, and conditionally write and move tape based on what is on tape. So if an LLM had access to infinite memory, all it needs is to implement a subleq instruction which is quite simple, and run itself in a loop. See https://en.m.wikipedia.org/wiki/One-instruction_set_computer One…
Read the full comment
instruction set computer See this paper from DeepMind which talks about compiling existing code into transformer primitives https://arxiv.org/pdf/2301.05062.pdf So the smallest ML model that can do arbitrary math with () * / + - operators and floating/int numbers is not that complex. Perhaps few 1000s of params. One can compile existing code into transformer weights. Now that you mention, I’ve been nerdsniped to building it.
Original comment Phi-2: The surprising power of small language models
teddykoker
Neutral mention · rule
A related line of work is "Thinking Like Transformers" [1]. They introduce a primitive programming language, RASP, which is composed of operations capable of being modeled with transformer components, and demonstrate how different programs can be written with it, e.g. histograms, sorting. Sasha Rush and Gail Weiss have an excellent blog post on it as well [2]. Follow on work actually demonstrated how RASP-like programs could actually be compiled into model weights without training [3]. [1] https://arxiv.org/abs/2106.06981 [2] https://srush.github.io/raspy/ [3] https://arxiv.org/abs/2301.05062
Original comment I made a transformer by hand (no training)
rslice
Neutral mention · rule
Perhaps GPT-like models are already capable enough to do math, but they need to store what we call mathematical reasoning as one of many distinct processing pathway and tap into it whenever the context is appropriate. Easy to say obviously but there's some promising work in this direction[1] [1]Tracr: Compiled Transformers as a Laboratory for Interpretability, https://arxiv.org/pdf/2301.05062.pdf
Original comment Why Can't Little Chatty Do Math?

Keep following the thread.

More in Language models

More from the same topics.