Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at https://aka.ms/LeX-Transformer.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
Introducing LLongMA, a series of OpenLLaMA models, trained at 8k context length using linear positional interpolation scaling. The model was trained in collaboration with theemozilla of NousResearch and Kaiokendev.
We worked directly with Kaiokendev, to extend the context length of the open-llama 7b and 3b models through fine-tuning. The fine-tuned models maintain the same perplexity at 8k extrapolation surpassing the performance of other recent methodologies.
Applying the method to the rotary position embedding requires only slight changes to the model's code by dividing the positional index, t, by a scaling factor.
You can review the benchmarks of the LLongMA models…
Read the full comment
trained at 8k when compared to the original Open-LLaMA models trained at 2k. We show slight performance boosts on multiple benchmarks and minimal degradation on others.
The LLongMA 7b model is available on huggingface to use:https://huggingface.co/conceptofmind/LLongMA-7b
The LLongMA 3b model can also be found on huggingface here:https://huggingface.co/conceptofmind/LLongMA-3b
The repository containing theemozilla’s implementation of scaled rotary embeddings can be found here:https://github.com/jquesnelle/scaled-rope
A LLongMA-13b model trained at 8k context length will be released soon. As well as a suite of LLongMA models trained at 16k and 32k context lengths.
If you would like to learn more about scaling rotary embeddings, I would strongly recommend reading Kaiokendev's blog posts on his findings:https://kaiokendev.github.io/
A PR to add scaled rotary embeddings to huggingface transformers is currently in progress: https://github.com/huggingface/transformers/pull/24653
The model was trained for ~1 billion tokens on togethercompute's Red Pajama dataset. The context length of the examples varies: https://huggingface.co/datasets/togethercomputer/RedPajama-D...
The pre-tokenized dataset will be available here for you to use soon: https://huggingface.co/datasets/conceptofmind/rp-packed-8k-n...
I would also recommend checking out the phenomenal research by OfirPress on ALiBi which laid the foundation for many of these scaling techniques: https://arxiv.org/abs/2108.12409
OfirPress just live-streamed his Ph.D. defense recently here:https://twitter.com/OfirPress/status/1677025700560388098?s=2...
It is also worth reviewing the paper, A Length-Extrapolatable Transformer, and xPos technique which also applies scaling to rotary embeddings: https://arxiv.org/pdf/2212.10554.pdf
We previously trained the first publicly available model with rotary embedding scaling here: https://twitter.com/EnricoShippole/status/165559930145459404...
The compute for this model release is all thanks to the generous sponsorship by carperai, EMostaque, and StabilityAI.
You can find out more about the NousResearch organization here:https://huggingface.co/NousResearch
The base OpenLLaMA models used in these experiments were developed by younggeng and haoliuhl and can be found here: https://huggingface.co/openlm-research
This is not an official StabilityAI or NousResearch product.
Credits to Zanzibased for the llama image.
If you have any questions about the data or models be sure to reach out and ask! I will try to respond promptly.
My hypothesis for why this works that it mitigates the downsides of rope
to eli5:
rope is the modern strategy used to give information to the model about how far a query and a key are apart when doing attention. It's the best strategy we have now, but has a major downside, where it makes some connections between tokens that are far apart much stronger than you would like them to be. Xpos (https://arxiv.org/pdf/2212.10554) is another paper by microsoft tackling issues with rope and you can see figure 1 on page 4 to get a visual interpretation of the sinusoidal attention strength (you would like it to be smooth).
I think a big reason differential transformers is working so…
Read the full comment
well, especially on long sequence stuff, because when both q1 and q2 don't match a token, the rope relative strength will still have the same value and the noise will cancel out. Leaving intended matches, but at the cost of somewhat dampening the original value rope brought.
Just a hypothesis though. It would be easy to test by running this experiment against a baseline where both use alibi attention (https://arxiv.org/pdf/2108.12409) which has a different set of tradeoffs this wouldn't mitigate, but still a really interesting result.
It could be done in a dozen ways. One beautiful method is just using the xPos positional embedding pioneered by Microsoft and scale the context window size at runtime (even better if your attention is subquadratic - again there is a dozen of varieties to pick from), see:
"A Length-Extrapolatable Transformer"
https://arxiv.org/abs/2212.10554
"Language Is Not All You Need: Aligning Perception with Language Models"
https://arxiv.org/abs/2302.14045
Notably, this positional embedding has been implemented by lucidrains in his x-transformers package: https://github.com/lucidrains/x-transformers/blob/main/x_tra...
I like this feature they are working on
https://arxiv.org/abs/2212.10554
as I'd say the most obvious limitation of today's transformers is the limited attention window. If you want ChatGPT to do a good job of summarizing a topic based on the literature the obvious thing is to feed a bunch of articles into it and ask it to summarize (how can you cite a paper you didn't read?) and that requires looking at maybe 400,000 - 4,000,000 tokens.
Similarly there is a place for a word embedding, a sentence embedding, a paragraph embedding, a chapter embedding, a book embedding, etc. but these have to be scalable and obviously the book embedding is bigger but I ought to be able to turn a query into a sentence embedding and somehow match it against larger document embeddings.
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.