Skip to content
The methodOpen data
Back to the collection

Emergent Representations of Program Semantics in Language Models Trained on Programs

Charles Jin, Martin Rinard

2023 · arXiv

Language modelsStatistics & probabilityPsychology
Read the paper
HN score
0
Mentions
5
Readers
3
Months
4
What’s in a score?

From the abstract

We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code. Our code is available at https://github.com/charlesjin/emergent-semantics.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

altruios
Neutral mention · rule
https://www.youtube.com/watch?v=uRDBco-cSK4 https://www.technologyreview.com/2024/11/27/1107377/a-minecr... sadly paid-walled (and decently coded) article. The point is, there appears to be emergent behavior happening: the forming of religion, governance, and social interaction that was not in any system-card. Which makes sense, A brain in a vat with no external stimuli is a boring thing indeed, so too would an LLM be outside of a harness and any input. But put either in a society... More generally: if having an internal model of how the world works improves next token prediction, then the LLM that contains that would be selected for. For humans, that is the case for our…
Read the full comment
predictions of what would come next (we don't use tokens, of course). We have other studies indicating that there appear to be internal representations of concepts: https://arxiv.org/abs/2305.11169 I really do want to push back on the LLM + harness = chatbot position. A chatbot really is just a served LLM, sure there is code around it that may look like a harness: but unless it loops, streams, or has an interval activation, then it is decidedly not a 'harness'. A harness is defined by having such a loop/stream/interval: regardless of tool-use. Agent means it acts agentically: which requires that loop/stream/interval. Harnesses are what turn chat-bots into (possible) agents.
Original comment If your agent commits a crime, who is responsible?
famouswaffles
Neutral mention · rule
You are making a lot of assumptions that are mostly wrong. Case in point: https://arxiv.org/abs/2305.11169 I'm asking for something testable, not some post-hoc rationalization you believe to be true. I'm not asking you to tell me how you think LLMs work. I'm asking you to define "real reasoning" such that i can test people and LLMs for it and distinguish "real reasoning" from "fake reasoning". This definition should include all humans while excluding all LLMs. If it cannot, then it's just an arbitrary distinction.
Original comment Markov chains are funnier than LLMs
og_kalu
Neutral mention · rule
>But to know that, one has to understand what the data means semantically. And our current AIs ... well, don't. Another wrong statement, you're on a roll today. https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2306.12672 There's a word we would use to describe your confidently erroneous statements were it one of the outputs an LLM. Wonder what that might be..
Original comment The bigger-is-better approach to AI is running out of road
og_kalu
Neutral mention · rule
>Actual reasoning, or reconstruction of existing texts containing similar reasoning? The papers were linked in another comment. 3 of them don't even have anything to do with a existing dataset testing. so yeah, actual. for the world model papers https://arxiv.org/abs/2210.13382 https://arxiv.org/abs/2305.11169 >Lack of access to cameras or vehicle controls isn't why it can't drive a car. It would be best to wait till what you say can be evaluated. that is your hunch, not fact. >The existence of numerous ChatGPT jailbreaks is evidence to the contrary. No it's not. People fall for social engineering and do what you ask. if you think people can't be easily derailed, boy do i…
Read the full comment
have a bridge for you. >Many people are of below average intelligence, or give up when something is hard but not impossible. Ok. Doesn't help your point. and many above average people don't reach expert level either. If you want to rationalize all that as "gave up when it wasn't impossible", go ahead lol but reality paints a very different picture. >If you have one machine that will make one attempt to solve a problem a day and succeeds 90% of the time and another that will make a billion attempts to solve a problem a second and succeeds 10% of the time, which one has solved more problems by the end of the week? "Problems" aren't made equal. Practically speaking, it's very unlikely the billion per second thinker is solving any of the caliber of problems the one attempt per day is solving. Solving more "problems" does not make you a super intelligence.
Original comment Governance of Superintelligence
og_kalu
Neutral mention · rule
No world model ? A world model is so obvious papers like these are more confirmation than surprise https://arxiv.org/abs/2305.11169 https://arxiv.org/abs/2210.13382 There a certain sentiment that AGI however you wish to define it won't infact be a "We'll know it when we see it" situation but rather a "AGI will arrive long before consensus reaches its AGI". LLMs have made me believe this will 100% be the case, either way. It's one thing to argue over things we can't evaluate even now but man the 100th "They can't reason!" every week is pretty funny when you can basically take your pick of reasonong type - Algorithmic, Casual, Inference, Analogical and read a paper showing strong performance. https://arxiv.org/abs/2212.09196 https://arxiv.org/abs/2305.00050 https://arxiv.org/abs/2204.02329 https://arxiv.org/abs/2211.09066 People refuse to see even what is staring right at them.
Original comment Just Calm Down About GPT-4 Already

Keep following the thread.

More in Language models

More from the same topics.