Hopfield Networks is All You Need

tricks that made these work and now being very strong competitors to Transformers. The key here is the combination of an linear associative memory (aka Hopfield Network, aka Fast Weight Programmer, aka State Expansion...) and pushing it into a sequence with gating similar to the original LSTM (input, forget, output gate) - while here this is only dependent on the current input not the previous state for linearity. The linearity is needed to make it sequence-parallelizable, there are efforts now to add non-linearities again, but let's see. Their main benefit+downside both is that they have a fixed-size state, and therefore linear (vs Transformer-quadratic) time complexity. For larger sizes they have become popular in hybrids with Transformer (Attention) Blocks, as there are problems with long context tasks [14]. Cool thing is they can also be distilled from pre-trained Transformers with not too much performance drop [16]. 3. Along the sequence dimension most things can be categorized in these two. Attention and Linear (Associative Memory Enhanced) RNNs are heavily using Matrix Multiplications and anything else would be a waste of FLOPs on current GPUs. The essence is how to store information and how to interact with it, there might be still interesting directions as other comments show. Other important topics that go into the depth / width of the model are: Mixture of Experts, Iteration (RNNs) in Depth[17]. Disclaimer: I'm author of xLSTM and we recently released a 7B model [18] trained at NXAI, currently the fastest linear RNN at this scale and performance. Happy to answer more questions on this or the current state in this field of research. [1] https://arxiv.org/abs/2008.02217 [2] https://arxiv.org/abs/2006.16236 [3] https://arxiv.org/pdf/2102.11174 [4] https://github.com/BlinkDL/RWKV [5] https://arxiv.org/abs/2307.08621 [6] https://arxiv.org/pdf/2312.00752 [7] https://arxiv.org/abs/2312.06635 [8] https://arxiv.org/pdf/2402.19427 [9] https://arxiv.org/abs/2402.18668 [10] https://arxiv.org/abs/2405.04517 [11] https://arxiv.org/abs/2407.04620 [12] https://arxiv.org/abs/2412.06464 [13] https://arxiv.org/abs/2501.00663 [14] https://arxiv.org/abs/2406.07887 [15] https://arxiv.org/abs/2405.04434 [16] https://arxiv.org/abs/2410.10254 [17] http://arxiv.org/abs/2502.05171 [18] https://huggingface.co/NX-AI/xLSTM-7b

theoretically derive the connection between recurrence and attention" but I couldn't find a longer theorem-proof section. So either this is done only in a cursory manner, or the proof is very easy. Recurrence and attention have been around for a long time as concepts, so surely there are already proofs in similar contexts of this fact (I am not working in this particular area of Machine Learning, so I don't know the SOTA by heart, but I strongly suspect that these aspects have been discusses previously; thr Hopfield Network paper I linked to unearthes some theoretical facts about attention, for example). So -based on my very cursory reading- this paper seems like an interesting approach, but I do see some holes in thr execution. Time will tell whether Rentetive Network will become mainstream or not. Ok, this was my five minute review of the paper. Now I have to urgently return to completing my actual reviews for NeurIPS, haha.

large companies to sidestep licensing restrictions by encoding whatever data/code they want to use into some neural network and then sell it as AI. For why these things are essentially mangled compression engines one can take a look at "Hopfield Networks is all you need": https://arxiv.org/abs/2008.02217. It allows representing all modern transformer networks (which is what CoPilot is using) as a bunch of hopfield networks which are essentially memory modules connected in some complicated topology to encode some data set.

well as potentially significant efficiency gains. Hopfield Networks is All You Need [3] analyzes the Transformer architecture as the classical Hopfield Network. This one got a lot of buzz on HN so I won't talk about it too much, but it's part of a slew of other analyses of the Transformer that basically show how generalizable the attention mechanism is. It also sorta confirms many researchers' inkling that Transformers are likely just memorizing patterns in their training corpus. Edit: Adding a few interesting older NLP papers that I came across this year. StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding [4] Do Syntax Trees Help Pre-trained Transformers Extract Information? [5] Learning to Compose Neural Networks for Question Answering [6] Parsing with Compositional Vector Grammars [7] [1] https://arxiv.org/abs/2006.11287 [2] https://arxiv.org/abs/2006.12878 [3] https://arxiv.org/abs/2008.02217 [4] https://arxiv.org/abs/1908.04577 [5] https://arxiv.org/abs/2008.09084 [6] https://arxiv.org/abs/1601.01705 [7] https://www.aclweb.org/anthology/P13-1045/