Skip to content
The methodOpen data
Back to the collection

Relating transformers to models and neural representations of the hippocampal formation

James C. R. Whittington, Joseph Warren, Timothy E. J. Behrens

2021 · arXiv

NeuroscienceLanguage modelsMachine learning
Read the paper
HN score
1
Mentions
6
Readers
5
Months
6
What’s in a score?

From the abstract

Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

triclops200
Recommendation · rule
As a researcher in these fields: this reasoning is tired, overblown, and just wrong. We have a lot of understanding of how the brain works overall. You don't. Go read the active inference book by Friston et. al. for some of the epistemological and behavioral mechanics (Yes, this applies to llms as well, they easily satisfy the requirements to be considered the mathematical object described as a markov blanket). And, yes, if you could somehow freeze a human's current physical configuration at some time, you would absolutely, in principle, given what we know about the universe, be able to concretely map input to into actions. You cannot separate a human's representative…
Read the full comment
configuration from their environment in this way, so, behavior appears much more non-deterministic. Another paper by Friston et al (Path Integrals, particular kinds, and strange things) describes systems much like modern modeling and absolutely falls under the same action minimization requirements for the math to work given the kinds of data acquisition, loss functions, and training/post-training we're doing as a research society with these models. I also recommend https://arxiv.org/abs/2112.04035, but, in short, transformer models have functions and emergent structures provably similar both empirically and mathematically to how we abstract and consider things. Along with https://arxiv.org/pdf/1912.10077, these 4 sources, alone, together strongly rebuke any idea that they are somehow not capable of learning to act like and think like us, though there's many more.
Original comment The Case That A.I. Is Thinking
soraki_soladead
Neutral mention · rule
You might enjoy this paper[0] which shows that recurrent position encodings recover grid cell representations and maps to path integration found in a popular model of the hippocampus. This isn't terribly surprising since RNNs have shown this before[1, 2] but its an interesting connection. [0] https://arxiv.org/abs/2112.04035 [1] https://arxiv.org/abs/1803.07770 [2] https://www.nature.com/articles/s41586-018-0102-6
Original comment A most profound video game: a good cognitive aid for research
utopcell
Neutral mention · rule
Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version.`` [1] https://arxiv.org/abs/2112.04035
Original comment Understand how transformers work by demystifying the math behind them
macrolime
Neutral mention · rule
The neuromatch computational neuroscience course also seems quite interesting, though maybe less of practical use. https://compneuro.neuromatch.io/ Recent research like "Relating transformers to models and neural representations of the hippocampal formation" might make it more relevant though (https://arxiv.org/abs/2112.04035v2) quote from the abstract of that paper: "Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show…
Read the full comment
that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension."
Original comment Ask HN: How to get back into AI?
gfodor
Neutral mention · rule
That’s just begging the question. We don’t know if DALL-E is creative in ways isomorphic to humans. It might be. How would one go about structuring a hypothesis? Have you read the original DALL-E paper? Worth considering: https://arxiv.org/abs/2112.04035
Original comment Playing with DALL-E 2

Keep following the thread.

More in Neuroscience

More from the same topics.