The Platonic Representation Hypothesis

Platonic representation hypothesis? https://arxiv.org/abs/2405.07987 Maybe in the same way fourier/wavelet basis is just the most natural way to work with certain signals or images, there's a certain representation for language and a representation for "thinking" that's natural. Maths could itself be said to be an abstract representation of the latter, and even if the idea of a "universal grammar" is dead, maybe there's still some natural space to work with all human languages owing to the shared biological priors we have as humans.

Fair push back, but I do think the LSTM vs Transformers point kinda supports my position in the limit, not refutes. Once the compute bottleneck is removed, LSTMs scale favourably. https://arxiv.org/pdf/2510.02228 (I believe there's similar work done on vanilla LSTMs, but I'd have to go digging) So the bottleneck was compute. Which is compatible with 'data or compute'. But to accept your point, at the time the algorothmic advances were useful/did unlock/remove the bottleneck. A wider point is that eventually (once compute and data are scaled enough) the algorithms are all learning the same representations: https://arxiv.org/pdf/2405.07987 And of course the canon: https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dat... http://www.incompleteideas.net/IncIdeas/BitterLesson.html Scaling compute & data > algorithmic cleverness

https://arxiv.org/abs/2505.12540 https://arxiv.org/abs/2405.07987 These, other papers, and the lottery ticket phenomenon; what it boils down to is that any neural network like system which encodes some common mapping of a phenomenon in the context of the world - not necessarily a world model, but some "real-world thing" - will tend to map to a limited number of permutations of some archetypal representation, which will resemble other mappings of the same thing. The lottery ticket phenomenon is a bit like the birthday paradox; there will be some number of structures in a large, random initialization of neural network weights that coincide with one or more archetypal mappings of complex objects. Some sub-networks are also useful mappings to features of one or more complex objects, which makes learning hierarchical nested networks of feature mappings easier; it's also why interpretability is so damned difficult.

> They're going to diverge if the underlying manner in which data gets memorized and encoded, such as with RNNs, like RWKV. In the original paper (https://arxiv.org/abs/2405.07987) the authors also compared the representations of transformer-based LLMs to convolution-based image models. They found just as much alignment between them as when both models were transformers.

Same hat, except 18 months later, assuming it survives peer review, reproduction, etc. (or: "The newer one proposes evidence that appears to support the older one.") https://arxiv.org/abs/2405.07987

Also related: https://arxiv.org/abs/2405.07987 As a resident Max Stirner fan, the idea that platonism is physically present in reality and provably correct is upsetting indeed.

This is called the The Platonic Representation Hypothesis https://arxiv.org/abs/2405.07987 We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is driving toward a shared statistical model of reality, akin to Plato's concept of an ideal reality. We term such a representation the platonic representation and discuss several possible selective pressures toward it. Finally, we discuss the implications of these trends, their limitations, and counterexamples to our analysis.

>I am not a neuroscientist, but I think it's likely that LLMs (with 10s of billions of parameters) and the human brain (with 1-2 orders of magnitude more neural connections[1]) process language in analogous ways Related is the platonic representation hypothesis where models apparently converge to similar representations of relationships between data points. https://phillipi.github.io/prh/ https://arxiv.org/abs/2405.07987

We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is driving toward a shared statistical model of reality, akin to Plato’s concept of an ideal reality. We term such a representation the platonic representation and discuss several possible selective pressures toward it. Finally, we discuss the implications of these trends, their limitations, and counterexamples to our analysis. https://arxiv.org/html/2405.07987v5

The use of this sort of anthropomorphic and "incantation" style prompting is a workaround while mechanistic interpretability and monosemanticity work[1] is done to expose the neuron(s) that have larger impacts on model behavior -- cf Golden Gate Claude. Further, even if end-users only have access to token input to steer model behavior, we likely have the ability to reverse engineer optimal inputs to drive desired behaviors; convergent internal representations[2] means this research might transfer across models as well (particularly, Gemma -> Gemini, as I believe they share the same architecture and training data). I suspect we'll see understandable super-human prompting (and higher-level control) emerge from GAN and interpretability work within the next few years. [1]: https://transformer-circuits.pub/2024/scaling-monosemanticit... [2]: https://arxiv.org/abs/2405.07987

Neural network representation spaces seem to converge, regardless of architecture: https://arxiv.org/abs/2405.07987 It would make sense for the human mental latent spaces to also converge. The reason is that the latent space exists to model the environment, which is largely shared among humans.

Thanks! Yeah, it seems like a lot can be done with just text while we wait for multimodal models to catch up. The recent Platonic Representation Hypothesis [1] also suggests that different models, regardless of modality, build the same internal representations of the world. [1] https://arxiv.org/abs/2405.07987

In fact, it looks like the model is doing those things internally. We argue that representations in AI models, particularly deep networks, are converging. First, we survey many examples of convergence in the literature: over time and across multiple domains, the ways by which different neural networks represent data are becoming more aligned. Next, we demonstrate convergence across data modalities: as vision models and language models get larger, they measure distance between datapoints in a more and more alike way. We hypothesize that this convergence is driving toward a shared statistical model of reality, akin to Plato’s concept of an ideal reality. We term such a representation the platonic representation and discuss several possible selective pressures toward it. Finally, we discuss the implications of these trends, their limitations, and counterexamples to our analysis. https://arxiv.org/html/2405.07987v5