Skip to content
The methodOpen data
Back to the collection

Efficient Streaming Language Models with Attention Sinks

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis

2023 · arXiv

Language models
Read the paper
HN score
0
Mentions
6
Readers
6
Months
3
What’s in a score?

From the abstract

Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a "sink" even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

6 selected excerpts · 6 total mentions

kingstnap
Neutral mention · rule
There are many many problems with attention. The softmax has issues regarding attention sinks [1]. The softmax also causes sharpness problems [2]. In general this decision boundary being Euclidean dot products isn't actually optimal for everything, there are many classes of problem where you want polyhedral cones [3]. Positional embedding are also janky af and so is rope tbh, I think Cannon layers are a more promising alternative for horizontal alignment [4]. I still think there is plenty of room to improve these things. But a lot of focus right now is unfortunately being spent on benchmaxxing using flawed benchmarks that can be hacked with memorization. I think a really…
Read the full comment
promising and underappreciated direction is synthetically coming up with ideas and tests that mathematically do not work well and proving that current arhitectures struggle with it. A great example of this is the VITs need glasses paper [5], or belief state transformers with their star task [6]. The Google one about what are the limits of embedding dimensions also is great and shows how the dimension of the QK part is actually important to getting good retrevial [7]. [1] https://arxiv.org/abs/2309.17453 [2] https://arxiv.org/abs/2410.01104 [3] https://arxiv.org/abs/2505.17190 [4] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5240330 [5] https://arxiv.org/abs/2406.04267 [6] https://arxiv.org/abs/2410.23506 [6] https://arxiv.org/abs/2508.21038
Original comment 'Attention is all you need' coauthor says he's 'sick' of transformers
hedgehog
Neutral mention · rule
Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific. Edit: The "verbosity sink" name is inspired by the idea from the paper below although they're not actually at all the same thing. https://arxiv.org/abs/2309.17453
Original comment Task-specific LLM evals that do and don't work
visarga
Neutral mention · rule
Vanishing gradient was an issue for non-residual deep networks and vanilla RNNs. While the long context memory issues are along sequence dimension, not network depth. The problem could be some kind of instability of attention as it scales above 10k tokens. A recent paper suggests attention mechanism needs a default value (a "sink"), and its absence produces instability. https://arxiv.org/abs/2309.17453 Another paper says the middle part is lossy while the beginning and end are better attended.
Original comment MemGPT – LLMs with self-editing memory for unbounded context
spidersouris
Neutral mention · rule
The paper published by Xiao et al. (2023)[0] states that "a surprisingly large amount of attention score is allocated to the initial tokens, irrespective of their relevance to the language modeling task" (p. 2). Does that mean that task prefixes used for LLM generation (e.g. "translate: [sentence]") are actually attention sinks? Or are they not? I don't really understand what they mean by "irrespective of their relevance to the language modeling task." [0] https://arxiv.org/pdf/2309.17453.pdf
Original comment Attention Sinks in LLMs for endless fluency
TheJCDenton
Neutral mention · rule
The demo [3] seems very promising. "Their method cleverly exploits the LLMs' tendency to use initial tokens as "attention sinks" to anchor the distribution of attention scores. By caching initial tokens alongside recent ones, StreamingLLM restored perplexity and achieved up to 22x faster decoding than prior techniques." [1] "We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more." [2] "we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment." [2] "StreamingLLM achieves an impressive speedup,…
Read the full comment
reaching up to 22.2× per token. Despite its reduced latency, StreamingLLM sustains a memory footprint consistent with the re-computation baseline." [2] [1] https://notes.aimodels.fyi/llm-infinite-context-window-strea... [2] https://arxiv.org/pdf/2309.17453.pdf [3] https://github.com/mit-han-lab/streaming-llm
Original comment StreamingLLM: Efficient streaming technique enable infinite sequence lengths

Keep following the thread.

More in Language models

More from the same topics.