Gabriel Orlanski, Devjeet Roy, Alexander Yun
Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, Aws Albarghouthi
2026 · arXiv
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
7 selected excerpts · 7 total mentions
For most of these benchmarks, I feel like a p50 and p95 (using SWE salary as a proxy?) human benchmark as reference would be interesting. Edit: FWIW the paper the post quoted has repositories as slop baseline https://arxiv.org/html/2603.24755v1#S4.SS2
no i'm spinning those up at some point this week. here's the first few prompts I used (claude opus 5 as the research orchestrator), (these were interspersed with lots of tools and assistant messages but it should get you kicked off. > fetch this article for slopcodebench and help me run an eval on a subset of problems with opus 5 https://arxiv.org/html/2603.24755v1 > Get all the context, fetch any mentioned repos, and then propose a plan to me. > i have an anthropic API key in .... > Let's do the three challenges with Opus 4.8 and Opus 5 and Fable please. I like your minimal set. Let's try it. What do you need from me? > Actually I changed my mind. I want to do two of the…
easy ones you picked and then I want you to pick the one with more checkpoints, maybe one of the harder ones, not the very hardest one but one with a higher number of checkpoints. > Actually let's do one easy, one medium, and one hard problem please. If we have a hard problem I'd like to see that. > lets rock - i think lets just do opus 4.8 and sonnet 5 and opus 5 since we our ZDR will block fable
Actually, this paper shows that the LLMs create a lot more slop and degradation of the code after the first prompt. https://arxiv.org/html/2603.24755v1. So iterating with the LLM will simply make it to produce worse and worse code.
Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense. In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1]. [1] https://arxiv.org/html/2603.24755
What feels already like old history is that Apple made a generous deal to OpenAI based on the premise that their AI could do the claims. Apple engineers spend months trying to prompt engineer their way, thinking the prompter is at fault if the soon to be AGI system diverged. Some of these instructions were trending out there, as reveals of how naive Apple was at the time. They could be traced from the device's logs so not so much of a leak: Don't hallucinate, strictly follow instructions, followed by all sort of refined predicates, appended as if an LLM had reason Then Apple released a paper to warn everyone (well, a few, and to save face) that we are getting fooled. https://ml-site.cdn-apple.com/papers/the-illusion-of-thinkin... In case Apple is a biased anti AI propagandist, here is a similar, more recent research paper from MIT and co: https://arxiv.org/html/2603.24755v1
These links were previously submitted to HN without much traction: Research Paper — https://arxiv.org/abs/2603.24755 SlopCodeBench — https://www.scbench.ai/ And of course, the XCancel version of this post: https://xcancel.com/godofprompt/status/2038335755995750758
This is the dumbest take there is about vibe coding. Claiming that managing complexity in a codebase doesn't matter anymore. I can't imagine that a competent engineer would come to the conclusion that managing complexity doesn't matter anymore. There is actually some evidence that coding agents struggle the same way humans do as the complexity of the system increases [0]. [0] https://arxiv.org/abs/2603.24755
More from the same topics.
Haoran Xu, Fredrik Kjolstad
Fast compilation is important when compilation occurs at runtime, such as query compilers in modern database systems and WebAssembly virtual machines in modern browsers.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.
Donald E. Knuth
The author presents two tricks to accelerate depth-first search algorithms for a class of combinatorial puzzle problems, such as tiling a tray by a fixed set of polyominoes.
Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan
Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups.