Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

The report is definitely worth reading although I don't think the data is strong enough to draw significant conclusions from. I personally think that this should be studied more in depth over a longer time period since the current study had a 50 hour threshold (not exactly but close enough). One thing which is mentioned in the research paper[1] is: > However, we see positive speedup for the one developer who has more than 50 hours of Cursor experience, so it’s plausible that there is a high skill ceiling for using Cursor, such that developers with significant experience see positive speedup. [1]: https://arxiv.org/pdf/2507.09089

The idea that AI will replace programmers has been around since the emergence of AI. I do not know what the future holds. But I know that using in AI in software engineering reduces productivity by almost 20%. My point is, one tries to distinguish fact from opinion. Source: https://arxiv.org/abs/2507.09089

https://arxiv.org/abs/2507.09089 “Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down.”

Those are some high profile (celebrity) developers. I wonder if they have measured their results? I believe that the perceived speed up of AI coding is often different from reality. The following paper backs this idea https://arxiv.org/abs/2507.09089 . Can you provide data that objects this view, based on these (celebrity) developers or otherwise?

It "obviously" does based on what, exactly? For most devs (and it appears you, based on your comments) the answer is "their own subjective impressions", but that METR study (https://arxiv.org/pdf/2507.09089) should have completely killed any illusions that that is a reliable metric (note: this argument works regardless of how much LLMs have improved since the study period, because it's about how accurate dev's impressions are, not how good the LLMs actually were).

For seasoned maintainers of open source repos, there is explicit evidence it does slow them down, even when they think it sped them up: https://arxiv.org/abs/2507.09089 Cue: "the tools are so much better now", "the people in the study didn't know how to use Cursor", etc. Regardless if one takes issue with this study, there are enough others of its kind to suggest skepticism regarding how much these tools really create speed benefits when employed at scale. The maintenance cliff is always nigh... There are definitely ways in which LLMs, and agentic coding tools scaffolded in top, help with aspects of development. But to say anyone who claims otherwise is either being disingenuous or doesn't know what they are doing, is not an informed take.

Unlike higher level programing languages AI doesn't actually make programmers more efficient (https://arxiv.org/abs/2507.09089). Many people who are great programmers and love programing aren't interested in having their role reduced to being QA where they just review the bad code AI designed and wrote all day long. In a hypothetical world where AI is actually decent enough to be any good at writing software, the demand for software being infinite won't save even one programmer's job because zero programmers will be needed to create any of it. Everyone who needs software will just ask AI to do it for them. Zero programing jobs needed ever again.

There's a few bits of information from the original sources that's left out: - The METR paper surveyed just 16 developers to arrive at their conclusion. Not sure how that got past review. [0] - The finding from the MIT report can also be viewed from a glass 5% full perspective: > Just 5% of integrated AI pilots are extracting millions in value. > Winning startups build systems that learn from feedback (66% of executives want this), retain context (63% demand this), and customize deeply to specific workflows. They start at workflow edges with significant customization, then scale into core processes. [1] [0] https://arxiv.org/abs/2507.09089 [1] https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...

"Just start iterating in chunks alongside the LLM". For those types of tasks it probably takes the same amount of time to form the idea without AI as with AI, this is what Metr found in its study of developer productivity. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... https://arxiv.org/abs/2507.09089

There's literally a study out the shows when developers think LLMs are making them 20% faster, it turned out to be making them 20% less productive: https://arxiv.org/abs/2507.09089

> Careful with the productivity claims, there's plenty of research showing that perspective is usually a distortion of some kind. I saw 1 study which suggested this.[1] Were there more? [1] https://arxiv.org/abs/2507.09089

The study you are alluding to is this one by METR (Model Evaluation & Threat Research): Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - https://arxiv.org/abs/2507.09089 """ Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%—AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). """

The study you referred to sounded super interesting, so I looked it up to read later. To save others a search, here is a blog post and the paper: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o... https://arxiv.org/abs/2507.09089 Thanks for mentioning it.

> It’s why the world wasted $10B+ on self driving car companies that obviously made no sense. Obviously... in what way? I feel the anti-ai pattern is clear. Self-driving cars don't work in my city so the whole concept is a hoax. LLMs don't code my proprietary language so it's a bubble. > From this study (https://arxiv.org/abs/2507.09089) I can tell this is going to be the most misquoted study in blogs and pop-sci books after the 10,000-hour mastery study. And it's just a preprint!

> There’s a strange disconnect in the industry. On one hand, GitHub claims that 20 million users are on Copilot, and Sundar Pichai says that over 25% of the code at Google is now written by AI. On the other hand, independent studies show that AI actually makes experienced developers slower. From the study[0]: > 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. This study continues to get a lot of airtime on HN and elsewhere. Folks probably should be skeptical of a study with the combination of a small number of subjects with a broad claim. [0] https://arxiv.org/pdf/2507.09089

Yet when you look beyond boilerplate code generation, it's not all that LLMs increase experienced developers productivity (even when they believe that it does): https://arxiv.org/abs/2507.09089 Edit: Hello downvoters, would love to know if you found any flawed argument, is this just because this study/comment contradicting the common narrative on HN or something else entirely?

So... Attack first? That what you're going with? DKE has been confirmed more than anything else. It was one of the few things not hit by the replication crisis. You're assessing how well you do, when aided or not, by a tool. That's still self assessment, I'm afraid. And that self-assessment is flawed. [0] [0] https://arxiv.org/abs/2507.09089