Skip to content
The methodOpen data
Back to the collection

NoLiMa: Long-Context Evaluation Beyond Literal Matching

Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt

Show all 7 authors

Trung Bui, Ryan A. Rossi, Seunghyun Yoon, Hinrich Schütze

2025 · arXiv

Language models
Read the paper
HN score
0
Mentions
8
Readers
3
Months
5
What’s in a score?

From the abstract

Recent large language models (LLMs) support long contexts ranging from 128K to 1M tokens. A popular method for evaluating these capabilities is the needle-in-a-haystack (NIAH) test, which involves retrieving a "needle" (relevant information) from a "haystack" (long irrelevant context). Extensions of this approach include increasing distractors, fact chaining, and in-context reasoning. However, in these benchmarks, models can exploit existing literal matches between the needle and haystack to simplify the task. To address this, we introduce NoLiMa, a benchmark extending NIAH with a carefully designed needle set, where questions and needles have minimal lexical overlap, requiring models to infer latent associations to locate the needle within the haystack. We evaluate 13 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 11 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. Our analysis suggests these declines stem from the increased difficulty the attention mechanism faces in longer contexts when literal matches are absent, making it harder to retrieve relevant information. Even models enhanced with reasoning capabilities or CoT prompting struggle to maintain performance in long contexts. We publicly release the dataset and evaluation code at https://github.com/adobe-research/NoLiMa.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

8 selected excerpts · 8 total mentions

consumer451
Neutral mention · rule
super-edit: Sorry, this is not a usage related question, I have move it to: https://news.ycombinator.com/item?id=47772971 Here is the question for which I cannot find an answer, and cannot yet afford to answer myself: In Claude Code, I use Opus 4.6 1M, but stay under 250k via careful session management to avoid known NoLiMa [0] / context rot [1] crap. The question I keep wanting answered though: at ~165k tokens used, does Opus 1M actually deliver higher quality than Opus 200k? NoLiMa would indicate that with a ~165k request, Opus 200k would suck, and Opus 1M would be better (as a lower percentage of the context window was used)... but they are the same model. However,…
Read the full comment
there are practical inference deployment differences that could change the whole paradigm, right? I am so confused. Anthropic says it's the same model [2]. But, Claude Code's own source treats them as distinct variants with separate routing [3]. Closest test I found [4] asserts they're identical below 200K but it never actually A/B tests, correct? Inside Claude Code it's probably not testable, right? According to this issue [5], the CLI is non-deterministic for identical inputs, and agent sessions branch on tool-use. Would need a clean API-level test. The API level test is what I really want to know for the Claude based features in my own apps. Is there a real benchmark for this? I have reached the limits of my understanding on this problem. If what I am trying to say makes any sense, any help would be greatly appreciated. If anyone could help me ask the question better, that would also be appreciated. [0] https://arxiv.org/abs/2502.05167 [1] https://research.trychroma.com/context-rot [2] https://claude.com/blog/1m-context-ga [3] https://github.com/anthropics/claude-code/issues/35545 [4] https://www.claudecodecamp.com/p/claude-code-1m-context-wind... [5] https://github.com/anthropics/claude-code/issues/3370
Original comment Claude Code Routines
ivzak
Neutral mention · rule
It seems to be the hit rate of a very straightforward (literal matching) retrieval. Just checked the benchmark description (https://huggingface.co/datasets/openai/mrcr), here it is: "The task is as follows: The model is given a long, multi-turn, synthetically generated conversation between user and model where the user asks for a piece of writing about a topic, e.g. "write a poem about tapirs" or "write a blog post about rocks". Hidden in this conversation are 2, 4, or 8 identical asks, and the model is ultimately prompted to return the i-th instance of one of those asks. For example, "Return the 2nd poem about tapirs". As a side note, steering away from the literal…
Read the full comment
matching crushes performance already at 8k+ tokens: https://arxiv.org/pdf/2502.05167, although the models in this paper are quite old (gpt-4o ish). Would be interesting to run the same benchmark on the newer models Also, there is strong evidence that aggregating over long context is much more challenging than the "needle extraction task": https://arxiv.org/pdf/2505.08140 All in all, in my opinion, "context rot" is far from being solved
Original comment HN discussion #47367526
daemonologist
Neutral mention · rule
I only ran the benchmark on Quasar Alpha*; the rest of the scores come from the original paper [0] which was published before 3.7 was available. This is a pretty expensive benchmark to run if you're paying for API usage - I'd actually originally set out to run it on Llama 4 but abandoned that after estimating the cost. * - I also reproduced the Llama 3.1 8B result to check my setup. [0] - https://arxiv.org/abs/2502.05167 / https://github.com/adobe-research/NoLiMa
Original comment Evidence suggesting Quasar Alpha is OpenAI's new model
consumer451
Neutral mention · rule
Hello Simon, sorry for asking about this tangent here, but have you seen this paper? Is it as important as it appears to be? Should this metric be on all system cards? > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. https://arxiv.org/abs/2502.05167
Original comment Show HN: My LLM CLI tool can run tools now, from Python code or plugins
consumer451
Neutral mention · rule
Related paper: https://arxiv.org/abs/2502.05167 > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. I post this because this information seems very important for users of LLMs, and devs implementing LLMs in their own solutions. The fall-off in accuracy is far faster and greater than I had imagined. Someone should really make this an ongoing thing, which evaluates new models as they are released. Or, this information should be included in all model system cards.
Original comment NoLiMa: Long-Context Evaluation Beyond Literal Matching
consumer451
Neutral mention · rule
This is a paper which echoes your experience, in general. I really wish that when papers like this one were created, someone took the methodology and kept running with it for every model: > For instance, the NoLiMa benchmark revealed that models like GPT-4o experienced a significant drop from a 99.3% performance rate at 1,000 tokens to 69.7% at 32,000 tokens. Similarly, Llama 3.3 70B's effectiveness decreased from 97.3% at 1,000 tokens to 42.7% at 32,000 tokens, highlighting the challenges LLMs face with longer contexts. https://arxiv.org/abs/2502.05167
Original comment GPT-4.1 in the API
consumer451
Neutral mention · rule
I am very curious how 3.7 and o1 pro perform in this regard: > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. https://arxiv.org/abs/2502.05167
Original comment OpenAI's o1-pro now available via API
consumer451
Neutral mention · rule
I just learned something about LLMs: start a new chat as often as possible. I knew that this was a best practice, but I didn't know how quickly the situation becomes dire. > We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%. [0] I copied all the code that I could from TFA and pasted it into OpenAI's tokenizer. It…
Read the full comment
counted ~15k tokens. Many other tokens were generated in TFA's chat, some of which are not visible to the user. I think it's fair to assume that the entire chat was at least 25k tokens, right? Therefore, I believe that by the end of that chat, 4o's performance was significantly degraded. I think a major skill to develop for LLM supported coding is to compress a chat after just few thousand tokens into something like a step1.md file. Then start a new chat with "read step1.md" as the first prompt, and so on.. Is my logic sound here? [0] https://arxiv.org/abs/2502.05167
Original comment Converting C to ASM to specs and then to a working Z/80 Speccy tape

Keep following the thread.

More in Language models

More from the same topics.