Skip to content
The methodOpen data
Back to the collection

Extending Context Window of Large Language Models via Positional Interpolation

Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian

2023 · arXiv

Language modelsAlgorithms
Read the paper
HN score
0
Mentions
5
Readers
5
Months
4
What’s in a score?

From the abstract

We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least $\sim 600 \times$ smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

filterfiber
Neutral mention · rule
Yes! The obvious answer is to just increase your positions and train for that. This requires a ton of memory however (context length is squared) so most are currently training at 4k/8k and then finetuning higher similar to many of the image models. However there's been some work that to "get extra milage" out of the current models so-to speak with rotary positions and a few other tricks. These in combination with finetuning is the current method many are using at the moment IIRC. Here's a decent overview https://aman.ai/primers/ai/context-length-extension/ Rope - https://arxiv.org/abs/2306.15595 Yarn (based on rope) - https://arxiv.org/pdf/2309.00071.pdf LongLoRA -…
Read the full comment
https://arxiv.org/pdf/2309.12307.pdf The bottleneck is quickly going to be inference. Since the current transformer models need the context length ^2, the memory requirements go up very quickly. IIRC a 4090 can _barely_ fit a 4bit 30B model in memory with 4096k context length. From my understanding some form of RNNs are likely to be the next step for longer context. See RWKV as an example of a decent RNN https://arxiv.org/abs/2305.13048
Original comment Is anyone using self hosted LLM day to day and training it like a new employee
ZeroCool2u
Neutral mention · rule
See: https://arxiv.org/abs/2306.15595 "We present Position Interpolation (PI) that extends the context window sizes of RoPE-based pretrained LLMs such as LLaMA models to up to 32768 with minimal fine-tuning (within 1000 steps), while demonstrating strong empirical results on various tasks that require long context, including passkey retrieval, language modeling, and long document summarization from LLaMA 7B to 65B. Meanwhile, the extended model by Position Interpolation preserve quality relatively well on tasks within its original context window. To achieve this goal, Position Interpolation linearly down-scales the input position indices to match the original context…
Read the full comment
window size, rather than extrapolating beyond the trained context length which may lead to catastrophically high attention scores that completely ruin the self-attention mechanism. Our theoretical study shows that the upper bound of interpolation is at least ∼600× smaller than that of extrapolation, further demonstrating its stability. Models extended via Position Interpolation retain its original architecture and can reuse most pre-existing optimization and infrastructure."
Original comment Llama 2 – Meta AI
moffkalast
Neutral mention · rule
Here's some more info on it: https://arxiv.org/pdf/2306.15595.pdf https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar... https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic... In short, the context is just an array of indexes passed along with the data, which can be changed to floats and encode more sparsely to scale to an arbitrarily small or large context. It does need some slight retraining of the model to work well though afaik. What's funnier is that Meta came up with it (that paper is theirs) and somehow didn't bother including it in LLama 2.
Original comment Llama 2 – Meta AI
knaik94
Neutral mention · rule
1) 7B foundational model means this is a base model that has not been fine-tuned on the "prompt: response:" structure, and it has 7 billion weights/biases. 2) Currently every model that can run locally was trained with a 2K context size. It's a hard limit on prompt length. There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras. This base model was trained with 8k sequences. 3. 1.5T tokens is the size of the total training corpus. Training cost and time scale increase with training size. [B] A. https://arxiv.org/abs/2306.15595 B. https://www.semianalysis.com/p/the-ai-brick-wall-a-practical... (Jan 2023)
Original comment XGen-7B, a new 7B foundational model trained on up to 8K length for 1.5T tokens

Keep following the thread.

More in Language models

More from the same topics.