Skip to content
The methodOpen data
Back to the collection

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Ofir Press, Noah A. Smith, Mike Lewis

2021 · arXiv

PsychologyLanguage modelsDatabases
Read the paper
HN score
1.5
Mentions
7
Readers
7
Months
5
What’s in a score?

From the abstract

Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training? We first show that extrapolation can be enabled by simply changing the position representation method, though we find that current methods do not allow for efficient extrapolation. We therefore introduce a simpler and more efficient position method, Attention with Linear Biases (ALiBi). ALiBi does not add positional embeddings to word embeddings; instead, it biases query-key attention scores with a penalty that is proportional to their distance. We show that this method trains a 1.3 billion parameter model on input sequences of length 1024 that extrapolates to input sequences of length 2048, achieving the same perplexity as a sinusoidal position embedding model trained on inputs of length 2048 but training 11% faster and using 11% less memory. ALiBi's inductive bias towards recency also leads it to outperform multiple strong position methods on the WikiText-103 benchmark.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

EnricoShippole
Strong recommendation · rule
Introducing LLongMA, a series of OpenLLaMA models, trained at 8k context length using linear positional interpolation scaling. The model was trained in collaboration with theemozilla of NousResearch and Kaiokendev. We worked directly with Kaiokendev, to extend the context length of the open-llama 7b and 3b models through fine-tuning. The fine-tuned models maintain the same perplexity at 8k extrapolation surpassing the performance of other recent methodologies. Applying the method to the rotary position embedding requires only slight changes to the model's code by dividing the positional index, t, by a scaling factor. You can review the benchmarks of the LLongMA models…
Read the full comment
trained at 8k when compared to the original Open-LLaMA models trained at 2k. We show slight performance boosts on multiple benchmarks and minimal degradation on others. The LLongMA 7b model is available on huggingface to use:https://huggingface.co/conceptofmind/LLongMA-7b The LLongMA 3b model can also be found on huggingface here:https://huggingface.co/conceptofmind/LLongMA-3b The repository containing theemozilla’s implementation of scaled rotary embeddings can be found here:https://github.com/jquesnelle/scaled-rope A LLongMA-13b model trained at 8k context length will be released soon. As well as a suite of LLongMA models trained at 16k and 32k context lengths. If you would like to learn more about scaling rotary embeddings, I would strongly recommend reading Kaiokendev's blog posts on his findings:https://kaiokendev.github.io/ A PR to add scaled rotary embeddings to huggingface transformers is currently in progress: https://github.com/huggingface/transformers/pull/24653 The model was trained for ~1 billion tokens on togethercompute's Red Pajama dataset. The context length of the examples varies: https://huggingface.co/datasets/togethercomputer/RedPajama-D... The pre-tokenized dataset will be available here for you to use soon: https://huggingface.co/datasets/conceptofmind/rp-packed-8k-n... I would also recommend checking out the phenomenal research by OfirPress on ALiBi which laid the foundation for many of these scaling techniques: https://arxiv.org/abs/2108.12409 OfirPress just live-streamed his Ph.D. defense recently here:https://twitter.com/OfirPress/status/1677025700560388098?s=2... It is also worth reviewing the paper, A Length-Extrapolatable Transformer, and xPos technique which also applies scaling to rotary embeddings: https://arxiv.org/pdf/2212.10554.pdf We previously trained the first publicly available model with rotary embedding scaling here: https://twitter.com/EnricoShippole/status/165559930145459404... The compute for this model release is all thanks to the generous sponsorship by carperai, EMostaque, and StabilityAI. You can find out more about the NousResearch organization here:https://huggingface.co/NousResearch The base OpenLLaMA models used in these experiments were developed by younggeng and haoliuhl and can be found here: https://huggingface.co/openlm-research This is not an official StabilityAI or NousResearch product. Credits to Zanzibased for the llama image. If you have any questions about the data or models be sure to reach out and ask! I will try to respond promptly.
Original comment HN discussion #36634454
chessgecko
Neutral mention · rule
My hypothesis for why this works that it mitigates the downsides of rope to eli5: rope is the modern strategy used to give information to the model about how far a query and a key are apart when doing attention. It's the best strategy we have now, but has a major downside, where it makes some connections between tokens that are far apart much stronger than you would like them to be. Xpos (https://arxiv.org/pdf/2212.10554) is another paper by microsoft tackling issues with rope and you can see figure 1 on page 4 to get a visual interpretation of the sinusoidal attention strength (you would like it to be smooth). I think a big reason differential transformers is working so…
Read the full comment
well, especially on long sequence stuff, because when both q1 and q2 don't match a token, the rope relative strength will still have the same value and the noise will cancel out. Leaving intended matches, but at the cost of somewhat dampening the original value rope brought. Just a hypothesis though. It would be easy to test by running this experiment against a baseline where both use alibi attention (https://arxiv.org/pdf/2108.12409) which has a different set of tradeoffs this wouldn't mitigate, but still a really interesting result.
Original comment Differential Transformer
Smerity
Neutral mention · rule
I think you've done a great explanation expansion except I believe it's ALiBi ("Attention with Linear Biases Enables Input Length Extrapolation"), a method of positional encoding (i.e. telling the Transformer model how much to weight a distant token when computing the current output token). This has been used on various other LLMs[2]. [1]: https://arxiv.org/abs/2108.12409 [2]: n.b. Ofir Press is co-creator of ALiBi https://twitter.com/OfirPress/status/1654538361447522305
Original comment ChatGPT for Teams
eriicar
Neutral mention · rule
Ofir Press is the creator of AliBi attention and Self-Ask prompting. Here are the papers that are discussed in the podcast: AliBi Attention: https://arxiv.org/abs/2108.12409 Self-Ask Prompting: https://arxiv.org/abs/2210.03350
Original comment HN discussion #37324715
malwrar
Neutral mention · rule
Curious, what led you to adjusting the parameters this way? Also, have you guys experimented with ALiBi[1] which claims better extrapolative results than rotary positional encoding? [1]: https://arxiv.org/abs/2108.12409 (charts on page two if you’re skimming)
Original comment Code Llama, an AI tool for coding
sanxiyn
Neutral mention · rule
No. As far as I know, they haven't said anything about this. Neither did OpenAI about gpt-4-32k. MosaicML did say something about MPT-7B-StoryWriter-65k+: https://www.mosaicml.com/blog/mpt-7b. They are using ALiBi (Attention with Linear Biases): https://arxiv.org/abs/2108.12409. I think OpenAI and Anthropic are using ALiBi or their own proprietary advances. Both seem possible.
Original comment Anthropic - Introducing 100K Token Context Windows, Around 75,000 Words
ofirpress
Neutral mention · rule
(I wrote ALiBi) You can read the paper here https://arxiv.org/abs/2108.12409 While intuitively it does seem like ALiBi would make it hard for the model to attend to things that are far away, in many scenarios we've tested with different models trained on different datasets, ALiBi always performs better than sinusoidal, rotary, and other embedding types, even when we're not using it to extrapolate to longer sequence lengths. These findings have been confirmed by others, including by the BLOOM open source LM project.
Original comment Replit's new Code LLM: Open Source, 77% smaller than Codex, trained in 1 week

Keep following the thread.

More in Psychology

More from the same topics.

2017arXiv

The Case for Learned Index Structures

Tim Kraska, Alex Beutel, Ed H. Chi +2

Indexes are models: a B-Tree-Index can be seen as a model to map a key to the position of a record within a sorted array, a Hash-Index as a model to map a key to a position of a record within an unsorted array, and a BitMap-Index as a model to indicate if a data record exists or not.

Databases
19 mentions14 months
5HN score