Skip to content
The methodOpen data
Back to the collection

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Albert Gu, Tri Dao

2023 · arXiv

Language modelsMachine learningBiology & medicine
Read the paper
HN score
0
Mentions
18
Readers
18
Months
11
What’s in a score?

From the abstract

Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5$\times$ higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

18 selected excerpts · 18 total mentions

cs702
Neutral mention · rule
> Who out there is working on ... infinite history? Many people are still working on improving RNNs, mostly in academia. Examples off the top of my head: * RWKV: https://arxiv.org/abs/2006.16236 / https://arxiv.org/abs/2404.05892 https://arxiv.org/abs/2305.13048 * Linear attention: https://arxiv.org/abs/2503.14456 * State space models: https://arxiv.org/abs/2312.00752 / https://arxiv.org/abs/2405.21060 * Linear RNNs: https://arxiv.org/abs/2410.01201 Industry OTOH has gone all-in on Transformers.
Original comment FlashAttention-T: Towards Tensorized Attention
godelski
Neutral mention · rule
> Something like GPT-2 I got 2 things to say here 1) There's plenty of things that can achieve similar performance to GPT-2 these days. We mentioned Mamba, they compared to GPT-3 in their first paper[0]. They compare with the open sourced version and you'll also see some other architectures referenced there like Hyena and H3. It's the GPT-Neo and GPT-J models. Remember GPT-3 is pretty much just a scaled up GPT-2. 2) I think you are underestimating the costs to train some of these things. I know Karpathy said you can now train GPT-2 for like $1k[1] but a single training run is a small portion of the total costs. I'll reference StyleGAN3 here just because the paper has good…
Read the full comment
documentation on the very last page[2]. Check out the breakdown but there's a few things I want to specifically point out. The whole project cost 92 V100 years but the results of the paper only accounted for 5 of those. That's 53 of the 1876 training runs. Your $1k doesn't get you nearly as far as you'd think. If we simplify things and say everything in that 5 V100 years cost $1k then that means they spent $85k before that. They spent $18k before they even went ahead with that project. If you want realistic numbers, multiply that by 5 because that's roughly what a V100 will run you (discounted for scale). ~$110k ain't too bad, but that is outside the budget of most small labs (including most of academia). And remember, that's just the cost of the GPUs, that doesn't pay for any of the people running that stuff. I don't expect you to know any of this stuff if you're not a researcher. Why would you? It's hard enough to keep up with the general AI trends, let alone niche topics lol. It's not an intelligence problem, it's a logistics problem, right? A researcher's day job is being in those weeds. You just get a lot more hours in the space. I mean I'm pretty out of touch of plenty of domains just because time constraints. > I don't think this is feasible for large investments. I may be wrong, but i also don't think other avenues aren't being funded. So I'm trying to say, I think your bar has been met. And I think if we are actually looking at the numbers, yeah, I do not think these avenues are being funded. But don't take it from me, take it from FeiFei Li[3] | Not a single university today can train a ChatGPT model I'm not sure if you're a researcher or not, you haven't answered that question. But I think if you were you'd be aware of this issue because you'd be living with it. If you were a PhD student you would see the massive imbalance of GPU resources given to those working closely with big tech vs those trying to do things on their own. If you were a researcher you'd also know that even inside those companies that there aren't much resources given to people to do these things. You get them on occasion like the StarFlow and TarFlow I pointed out before, but these tend to be pretty sporadic. Even a big reason we talk about Mamba is because of how much they spent on it. But if you aren't a researcher I'd ask why you have such confidence that these things are being funded and that these things cannot be scaled or improved[4]. History is riddled with examples of inferior tech winning mostly due to marketing. I know we get hyped around new tech, hell, that's why I'm a researcher. But isn't that hype a reason we should try to address this fundamental problem? Because the hype is about the advance of technology, right? I really don't think it is about the advancement of a specific team, so if we have the opportunity for greater and faster advancement, isn't that something we should encourage? Because I don't understand why you're arguing against that. An exciting thing of working at the bleeding edge is seeing all the possibilities. But a disheartening thing about working at the bleeding edge is seeing many promising avenues be passed by for things like funding and publicity. Do we want meritocracy to win out or the dollar? I guess you'll have to ask yourself: what's driving your excitement? [0] I mean the first Mamba paper, not the first SSM paper btw: https://arxiv.org/abs/2312.00752 [1] https://github.com/karpathy/llm.c/discussions/677 [2] https://arxiv.org/abs/2106.12423 [3] https://www.ft.com/content/d5f91c27-3be8-454a-bea5-bb8ff2a85... [4] I'm not saying any of this stuff is straight de fact better. But there definitely is an attention imbalance and you have to compare like to like. If you get to x in 1000 man hours and someone else gets there in 100, it may be worth taking a look deeper. That's all.
Original comment GPT-5: Key characteristics, pricing and system card
korbip
Neutral mention · rule
There is a LOT of effort in the research community currently: 1. Improving the Self-Attention in the Transformer as is, keeping the quadratic complexity, which has some theoretical advantage in principle[1]: The most hyped one probably DeepSeek's Multi-Latent Attention[15], which kind of is Attention still - but also somehow different. 2. Linear RNNs: This starts from Linear Attention[2], DeltaNet[3], RKWV[4], Retention[5], Gated Linear Attention[6], Mamba[7], Griffin[8], Based[9], xLSTM[10], TTT[11], Gated DeltaNet[12], Titans[13]. They all have an update like: C_{t} = F_{t} C_{t-1} + i_{t} k_{t} v_{t}^T with a cell state C and output h_{t} = C_{t}^T q_{t}. There's a few…
Read the full comment
tricks that made these work and now being very strong competitors to Transformers. The key here is the combination of an linear associative memory (aka Hopfield Network, aka Fast Weight Programmer, aka State Expansion...) and pushing it into a sequence with gating similar to the original LSTM (input, forget, output gate) - while here this is only dependent on the current input not the previous state for linearity. The linearity is needed to make it sequence-parallelizable, there are efforts now to add non-linearities again, but let's see. Their main benefit+downside both is that they have a fixed-size state, and therefore linear (vs Transformer-quadratic) time complexity. For larger sizes they have become popular in hybrids with Transformer (Attention) Blocks, as there are problems with long context tasks [14]. Cool thing is they can also be distilled from pre-trained Transformers with not too much performance drop [16]. 3. Along the sequence dimension most things can be categorized in these two. Attention and Linear (Associative Memory Enhanced) RNNs are heavily using Matrix Multiplications and anything else would be a waste of FLOPs on current GPUs. The essence is how to store information and how to interact with it, there might be still interesting directions as other comments show. Other important topics that go into the depth / width of the model are: Mixture of Experts, Iteration (RNNs) in Depth[17]. Disclaimer: I'm author of xLSTM and we recently released a 7B model [18] trained at NXAI, currently the fastest linear RNN at this scale and performance. Happy to answer more questions on this or the current state in this field of research. [1] https://arxiv.org/abs/2008.02217 [2] https://arxiv.org/abs/2006.16236 [3] https://arxiv.org/pdf/2102.11174 [4] https://github.com/BlinkDL/RWKV [5] https://arxiv.org/abs/2307.08621 [6] https://arxiv.org/pdf/2312.00752 [7] https://arxiv.org/abs/2312.06635 [8] https://arxiv.org/pdf/2402.19427 [9] https://arxiv.org/abs/2402.18668 [10] https://arxiv.org/abs/2405.04517 [11] https://arxiv.org/abs/2407.04620 [12] https://arxiv.org/abs/2412.06464 [13] https://arxiv.org/abs/2501.00663 [14] https://arxiv.org/abs/2406.07887 [15] https://arxiv.org/abs/2405.04434 [16] https://arxiv.org/abs/2410.10254 [17] http://arxiv.org/abs/2502.05171 [18] https://huggingface.co/NX-AI/xLSTM-7b
Original comment Ask HN: Is anybody building an alternative transformer?
viktor_von
Neutral mention · rule
> The information cost of making the RNN state way bigger is high when done naively, but maybe someone can figure out a clever way to avoid storing full hidden states in memory during training or big improvements in hardware could make memory use less of a bottleneck. Isn't this essentially what Mamba [1] does via its 'Hardware-aware Algorithm'? [1] https://arxiv.org/pdf/2312.00752
Original comment Were RNNs all we needed?
Fripplebubby
Neutral mention · rule
To further problematize this question (which I don't feel like I can actually answer), consider this paper: "Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention" - https://arxiv.org/pdf/2006.16236 What this shows is that actually a specific narrow definition of transformer (a transformer with "causal masking" - see paper) is equivalent to an RNN, and vice versa. Similarly Mamba (https://arxiv.org/abs/2312.00752), the other hot architecture at the moment, has an equivalent unit to a gated RNN. For performance reasons, I believe they use an equivalent CNN during training and an RNN during inference!
Original comment My Python code is a neural network
edflsafoiewq
Neutral mention · rule
The doc comment at the top of the .py file is sufficiently descriptive """Simple, minimal implementation of Mamba in one file of Numpy adapted from (1) and inspired from (2). Suggest reading the following before/while reading the code: [1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Albert Gu and Tri Dao) https://arxiv.org/abs/2312.00752 [2] The Annotated S4 (Sasha Rush and Sidd Karamcheti) https://srush.github.io/annotated-s4
Original comment A pure NumPy implementation of Mamba
sva_
Neutral mention · rule
It's an LLM architecture competing with transformers: https://arxiv.org/abs/2312.00752 Proponents of it usually highlight it's inference performance, in particular linear scaling with the input tokens.
Original comment A pure NumPy implementation of Mamba
fancyfredbot
Neutral mention · rule
A paper describing how you might do it published in December last year. The paper was "Mamba: Linear-Time Sequence Modeling with Selective State Spaces". To be clear I don't know if Claude and Gemini actually use this technique but I would not be surprised if they did something similar: https://arxiv.org/abs/2312.00752 https://github.com/state-spaces/mamba
Original comment Claude 3 model family
yuhongsun
Neutral mention · rule
Ya, a lot of knowledge is locked away in metadata, how documents are structured, in non-textual representations, or sometimes even just in the head of experts. We definitely want to be more than a shallow access to documents and we're building with that in mind. We're currently working to include more metadata and organizational understanding, with plans to tackle OCR, NL-to-SQL, code search and knowledge graphs in the future. While we don't have concrete dates for all of these, hopefully this gives some visibility into our vision at least! Regarding the context size, the research community is doing some stunning work. If you're interested, you should check out the new Mamba (Transformers replacement) architecture: https://arxiv.org/abs/2312.00752
Original comment Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data
jackcook
Neutral mention · rule
Yes you're spot on, the nonlinearities come from the full Mamba blocks, which I left out of this post for simplicity/to focus on the bigger ideas the paper introduced. You can see it marked by the "X" on the right-most part of Figure 3 in the Mamba paper: https://arxiv.org/abs/2312.00752
Original comment Mamba: The Easy Way
jxmorris12
Neutral mention · rule
In case people are wondering why Mamba is exciting: There's this idea in AI right now that "scaling" models to be bigger and train on more data always makes them better. This has led to a science of "scaling laws" which study just how much bigger models need to be and how much data we need to train them on to make them a certain amount better. The relationship between model size, training data size, and performance turns out to be quite predictable. Transformers are great because they can continue scaling and giving us better performance – unlike, we think, RNNs. Probably the most exciting thing about Mamba is the claim that it can be a bit smaller, and train on a bit less data, and still provide better performance than the equivalent Transformer, especially at longer sequence lengths. For more info, see the scaling laws plot in Figure 4 of the Mamba paper: https://arxiv.org/abs/2312.00752
Original comment Mamba: The Easy Way
TweedBeetle
Neutral mention · rule
Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture. Here’s the paper: https://arxiv.org/abs/2312.00752 And here’s a great podcast episode on it: https://www.cognitiverevolution.ai/emergency-pod-mamba-memor...
Original comment Our next-generation model: Gemini 1.5
sanxiyn
Neutral mention · rule
I agree that on balance, we should spend more effort on data than modeling, but it is just not true that modeling doesn't matter. Transformer-2023 is different from Trnasformer-2020 and cumulative improvement is significant. https://arxiv.org/abs/2312.00752 did such benchmark. If choice between Transformer and RWKV doesn't seem to matter to you, the only reason is that while Transformer-2020 evolved to Transformer-2023, RWKV-v1 (which is from 2021) also evolved to RWKV-v5. If you use Transformer-2020 or RWKV-v1 today you will feel the difference.
Original comment Eagle 7B: Soaring past Transformers
MAXPOOL
Neutral mention · rule
Mamba is a new model architecture based on SSM's. Mamba: Linear-Time Sequence Modeling with Selective State Spaces https://arxiv.org/abs/2312.00752 https://github.com/state-spaces/mamba Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model https://paperswithcode.com/paper/vision-mamba-efficient-visu...
Original comment Introduction to State Space Models (SSM)
squigz
Neutral mention · rule
This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the link and read discussion on reddit - if I wanted that sort of discussion, I would browse there, not HN.
Original comment Mamba outperforms transformers "everywhere we tried"

Keep following the thread.

More in Language models

More from the same topics.