Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

Theres a study from earlier this year which suggests the opposite: https://arxiv.org/abs/2602.11988 Most of the agents.md and what people use it for / write into is does, in fact, not make a difference. Now, sure, this study is a bit old for LLM standards - as everything beyond the current month is - but a) I haven't seen any tangible evidence to the contrary and b) Since the basic inner workings of LLMs haven't changed I'd be sceptical of this not still applying. I think one major side effect of LLMs moving so fast is that best practices and how to use this tool is very much not catching up as fast. No one knows what is best and what actually makes a difference, doubly so because LLMs are / very / hard to quantify - even benchmarks themselves are very rough estimations. People do, in fact, use stuff which makes no difference all the times.

Believe it or not, after writing this comment I was doing some more reading on the task. I'm planning to reorganize our context repo after finding this paper (it argues that AI generated context files can stunt the performance of models): https://arxiv.org/abs/2602.11988 For what it's worth, if you were considering building context out.

Are you referring to the one (1) study that showed that when cheaper LLM's auto-generated an AGENTS.md, it performed more poorly than human editted AGENTS.md? https://arxiv.org/abs/2602.11988 I'd love to see other sources that seek to academically understand how LLM's use context, specifically ones using modern frontier models. My takeaway from these CLAUDE.md/AGENTS.md efforts isn't that agents can't maintain any form of context at all, rather, that bloated CLAUDE.md files filled with data that agents can gather on the spot very quickly are counter-productive. For information which cannot be gathered on the spot quickly, clearly (to me) context helps improve quality, and in my experience, having AI summarize some key information in a thread and write to a file, and organize that, has been helpful and useful.

Great, so how do you know this stuff works? Did you evaluate it against other approaches? How do you know it's actually reliable? The Vercel team had some interesting findings[1]: > In 56% of eval cases, the skill was never invoked. The agent had access to the documentation but didn't use it. Others had different findings for commonly accepted practices[2], some you may have adopted from reading documentation, which surely didn't come from influencers. And yet others swear by magical Markdown documents[3]. So... who is the ultimate authority on what actually works, and who is just cargo culting the trendy practice of the week? And how is any of this different from what was being done a few years ago? [1]: https://vercel.com/blog/agents-md-outperforms-skills-in-our-... [2]: https://arxiv.org/abs/2602.11988 [3]: https://soul.md/

Another reference: https://arxiv.org/abs/2602.11988. Too many times they ignore what is defined depending on the prompt and/or context. It's just another task to sync a source of truth.

The research is: Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? https://arxiv.org/abs/2602.11988 --- Overall, it makes sense to me that excessive and/or unnecessary instructions dilute the importance of each instruction. Ideally it shouldn't, but it does.

This method of copying an LLM-generated summary of your preferences into Claude memory feels similar to their recommendation to use /init to generate a CLAUDE.md based on the project, which recent research[0] suggests may be counterproductive. I would assume both Claude memory and CLAUDE.md work best when they're carefully curated, only containing what you've found yourself having to repeat. [0]: https://arxiv.org/abs/2602.11988

ETH Zurich tested this: LLM-generated prose context = -3% performance, +20% cost. Even human-written = +4% at +19% cost. The problem is prose bloat. Structured formats avoid that by design. https://arxiv.org/abs/2602.11988

The persistent identity files are interesting but there's a cost problem. A recent paper (arxiv 2602.11988 https://arxiv.org/html/2602.11988v1) found context files increase inference cost by 20%+ with marginal performance gains; LLM-generated ones actually decreased success rates slightly. Four identity files per agent injected every session feels like monkey patching coherence with context. Context isn't memory, it's just more tokens. The hard unsolved problem is cross-session learning without the bloat. Curious if you've measured the token overhead of the identity files vs the performance gain they provide.

I'm actually on the fence with skills. Cursor shared a study where they claimed skills performed actually worse - than just injecting into the context directly. Similarly, there was a paper recently that suggested the same [1] https://arxiv.org/abs/2602.11988 Of course, the classic response to these - event WITH the evidence is often "yOu'Re dOiNg iT wRonG". Does anyone actually have proof - where using skill.md is arguably better than not?

There is some papers [0] showing that the skill and agent files reduce the reasoning effectiveness in some use cases (e.g. autogenerated) [0] https://arxiv.org/abs/2602.11988 reference: https://news.ycombinator.com/item?id=47034087