The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
> Now, we can see from this description that nothing about the modeling ensures that the outputs accurately depict anything in the world. There is not much reason to think that the outputs are connected to any sort of internal representation at all.
This is just wrong. Accurate modelling of language at the scale of modern LLMs requires these models to develop rich world models during pretraining, which also requires distinguishing facts from fiction. This is why bullshitting happens less with better, bigger models: the simple answer is that they just know more about the world, and can also fill in the gaps more efficiently.
We have empirical evidence here: it's even…
Read the full comment
possible to peek into a model to check whether the model 'thinks' what it's saying is true or not. From “Discovering Latent Knowledge in Language Models Without Supervision” (2022) [1]:
> Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations. It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values. (...) We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers. Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don't have access to explicit ground truth labels.
So when a model is asked to generate an answer it knows is incorrect, it's internal state still tracks the truth value of the statements. This doesn't mean the model can't be wrong about what it thinks is true (or that it won't try to fill in the gaps incorrectly, essentially bullshitting), but it does mean that the world models are sensitive to truth.
More broadly, we do know these models have rich internal representations, and have started learning how to read them. See for example “Language Models Represent Space and Time” (Wes & Tegmark, 2023) [2]:
> We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.
For anyone curious, I can recommend the Othello-GPT paper as a good introduction to this problem (“Do Large Language Models learn world models or just surface statistics?”) [3].
[1]: https://arxiv.org/abs/2310.02207
[2]: https://arxiv.org/abs/2310.02207
[3]: https://thegradient.pub/othello/
> They are literally next token prediction machines normally trained on just text tokens.
And in order to predict the next token well they have to build world models, otherwise they would just output nonsense. This has been proven [1].
This notion that just calling them "next token predictors" somehow precludes them being intelligent is based on a premise that human intelligence cannot be reduced to next token prediction, but nobody has proven any such thing! In fact, our best models for human cognition are literally predictive coding.
LLMs are probably not the final story in AGI, but claiming they are not reasoning or not understanding is at best speculation, because we lack a mechanistic understanding of what "understanding" and "reasoning" actually mean. In other words, you don't know that you are not just a fancy next token predictor.
[1] https://arxiv.org/abs/2310.02207
There's been a fair bit of research over the last year on this topic.
An easy read on a Harvard/MIT study: https://thegradient.pub/othello/
A follow-up on more technical aspects of what's going on with it: https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-o...
Two more studies since showing linear representations of world models:
https://arxiv.org/abs/2310.02207 (modeling space and time)
https://arxiv.org/abs/2310.06824 (modeling truth vs falsehood)
It's worth keeping in mind these are all on smaller toy models compared to something like GPT-4, so there's likely more complex versions of a similar thing going on there, we just don't know to what extent as it's a…
Read the full comment
black box.
Part of the problem with evaluating the models based on responses is that they are both surface statistics/correlations and deeper processing, and often the former can obscure the latter. For example, in the first few weeks of release commentators on here were pointing out GPT-4 failed at variations of the wolf, goat, and cabbage problem. And indeed, giving it a version with a vegetarian wolf and a carnivorous goat it would still go to the classic answer of taking the goat first. But if you asked it to always repeat adjectives and nouns from the original problem together and change the nouns to emojis (, , ), it got it right every single time on the first try. So it did have the capacity to reason out variations of the problem, you just needed to bust the bias towards surface statistics around the tokens first.
This varies a lot by people - some people's conscious thought is in images, some in language, some in unsymbolised concepts.
Hurlburt's "Descriptive Experience Sampling" is a good source on this - see for example table 2 of https://hurlburt.faculty.unlv.edu/heavey-hurlburt-2008.pdf
Agreed that underneath that perceivable layer, there's lots else going on too. But to be fair, there's also lots going on the weights of GPT-4, e.g. see https://arxiv.org/abs/2310.02207
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.