Ali Behrouz, Peilin Zhong, Vahab Mirrokni
2024 · arXiv
Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current context while utilizing long past information. We show that this neural memory has the advantage of fast parallelizable training while maintaining a fast inference. From a memory perspective, we argue that attention due to its limited context but accurate dependency modeling performs as a short-term memory, while neural memory due to its ability to memorize the data, acts as a long-term, more persistent, memory. Based on these two modules, we introduce a new family of architectures, called Titans, and present three variants to address how one can effectively incorporate memory into this architecture. Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
9 selected excerpts · 9 total mentions
don't miss the (formerly over-hyped) Google Research's Titans+Atlas thread, they are quite inspiring and informative on the subject; even if I don't think they released anything (open, at least) as a follow-up to their research. https://arxiv.org/abs/2501.00663 https://arxiv.org/abs/2505.23735
Putting stuff you have learned into a markdown file is a very "shallow" version of continual learning. It can remember facts, yes, but I doubt a model can master new out-of-distribution tasks this way. If anything, I think that Google's Titans[1] and Hope[2] architectures are more aligned with true continual learning (without being actual continual learning still, which is why they call it "test-time memorization"). [1] https://arxiv.org/pdf/2501.00663 [2] https://arxiv.org/pdf/2512.24695
Per your point 4, some current hyped work is pushing hard in this direction [1, 2, 3]. The basic idea is to think of attention as a way of implementing an associative memory. Variants like SDPA or gated linear attention can then be derived as methods for optimizing this memory online such that a particular query will return a particular value. Different attention variants correspond to different ways of defining how the memory produces a value in response to a query, and how we measure how well the produced value matches the desired value. Some of the attention-like ops proposed in this new work are most simply described as implementing the associative memory with a…
hypernetwork that maps keys to values with weights that are optimized at test time to minimize value retrieval error. Like you suggest, designing these hypernetworks to permit efficient implementations is tricky. It's a more constrained interpretation of attention than you're advocating for, since it follows the "attention as associative memory" perspective, but the general idea of test-time optimization could be applied to other mechanisms for letting information interact non-linearly across arbitrary nodes in the compute graph. [1] https://arxiv.org/abs/2501.00663 [2] https://arxiv.org/abs/2504.13173 [3] https://arxiv.org/abs/2505.23735
Have you looked at Titan and MIRAS where they use online/updating associative memory that happens to be read out via next-token prediction? https://research.google/blog/titans-miras-helping-ai-have-lo... https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Much research is going into these directions, but I'm more interested in mind-wandering tangents, involving both attentional control and additional mechanisms (memory retrieval, self-referential processing).
From the blog: https://arxiv.org/abs/2501.00663 https://arxiv.org/pdf/2504.13173 Is there any other company that's openly publishing their research on AI at this level? Google should get a lot of credit for this.
Something like this, using Simon Willison's llm cli, will give you a better experience than such generated articles unless your goal is explicitly to read blogs. wget https://arxiv.org/pdf/2501.00663 -O - | pdftotext - - | llm -s "Explain the following paper assuming I am familiar with deep learning" -m gpro Advantages: * put in any prompt you like * pick any model (above gpro is alias for latest gemini pro if you have the llm-gemini plugin installed) * ask for clarifications using llm chat -c (continue last conversation) * no annoying random image For another approach for this particular paper, Umal Jamil's videos are always high quality: https://www.youtube.com/watch?v=A6kPQVejN4o
The paper the article is based on [1] treats “Titans” as a plural: “Titans are capable of solving problems beyond TC^0, meaning that Titans are theoretically more expressive than Transformers….” [1] https://arxiv.org/pdf/2501.00663
There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t} k_{t} v_{t}^T with a cell state C and output h_{t} = C_{t}^T q_{t}. There's a few…
tricks that made these work and now being very strong competitors to Transformers. The key here is the combination of an linear associative memory (aka Hopfield Network, aka Fast Weight Programmer, aka State Expansion...) and pushing it into a sequence with gating similar to the original LSTM (input, forget, output gate) - while here this is only dependent on the current input not the previous state for linearity. The linearity is needed to make it sequence-parallelizable, there are efforts now to add non-linearities again, but let's see. Their main benefit+downside both is that they have a fixed-size state, and therefore linear (vs Transformer-quadratic) time complexity. For larger sizes they have become popular in hybrids with Transformer (Attention) Blocks, as there are problems with long context tasks [14]. Cool thing is they can also be distilled from pre-trained Transformers with not too much performance drop [16]. 3. Along the sequence dimension most things can be categorized in these two. Attention and Linear (Associative Memory Enhanced) RNNs are heavily using Matrix Multiplications and anything else would be a waste of FLOPs on current GPUs. The essence is how to store information and how to interact with it, there might be still interesting directions as other comments show. Other important topics that go into the depth / width of the model are: Mixture of Experts, Iteration (RNNs) in Depth[17]. Disclaimer: I'm author of xLSTM and we recently released a 7B model [18] trained at NXAI, currently the fastest linear RNN at this scale and performance. Happy to answer more questions on this or the current state in this field of research. [1] https://arxiv.org/abs/2008.02217 [2] https://arxiv.org/abs/2006.16236 [3] https://arxiv.org/pdf/2102.11174 [4] https://github.com/BlinkDL/RWKV [5] https://arxiv.org/abs/2307.08621 [6] https://arxiv.org/pdf/2312.00752 [7] https://arxiv.org/abs/2312.06635 [8] https://arxiv.org/pdf/2402.19427 [9] https://arxiv.org/abs/2402.18668 [10] https://arxiv.org/abs/2405.04517 [11] https://arxiv.org/abs/2407.04620 [12] https://arxiv.org/abs/2412.06464 [13] https://arxiv.org/abs/2501.00663 [14] https://arxiv.org/abs/2406.07887 [15] https://arxiv.org/abs/2405.04434 [16] https://arxiv.org/abs/2410.10254 [17] http://arxiv.org/abs/2502.05171 [18] https://huggingface.co/NX-AI/xLSTM-7b
The AI hardware race is still going strong, but with so many rapid changes to the fundamental architectures, it doesn't make sense to bet everything on specialized hardware just yet.. It's happening, but it's expensive and slow. There's just not enough capacity to build memory fast enough right now. Everyone needs the biggest and fastest modules they can get, since it directly impacts the performance of the models. There's still a lot of happening to improve memory, like the latest Titans paper: https://arxiv.org/abs/2501.00663 So I think until a breakthrough happens or the fabs catch up, it'll be this painful race to build more datacenters.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.