Skip to content
The methodOpen data
Back to the collection

How AI Impacts Skill Formation

Judy Hanwen Shen, Alex Tamkin

2026 · arXiv

Programming languagesNeuroscience
Read the paper
HN score
0
Mentions
5
Readers
4
Months
4
What’s in a score?

From the abstract

AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

simonw
Neutral mention · rule
The bit about computer science (behind the paywall) starts: > To investigate whether skills are being lost in the field of computer science, researchers at the AI firm Anthropic in San Francisco, California, designed a randomized controlled trial in which 52 software engineers were asked to perform a basic coding task That's this study here: https://arxiv.org/abs/2601.20245 - also written about on the Anthropic research site here: https://www.anthropic.com/research/AI-assistance-coding-skil...
Original comment Is AI ruining our skills? Early results are in – and they're not good
keeda
Neutral mention · rule
I hadn't looked at that study you selected, but yeah the methodology conflicts with the abstract (Also it low-key seems to be an ad for "Flexi 2.0.") It does seems to be a shady paper, with a small N and in a journal of questionable repute. That said, there are 80+ other studies listed in the meta-study, which is pretty frank about its limitations. (Note the snippets about positive biases in the conclusion.) It is going more for quantity over quality and is transparent about the statistical findings of each one (or lack thereof; see the count of "Not reported"s.) All these references have a myriad of results, but across the spectrum of well-designed studies at reputable…
Read the full comment
venues to the other end, they follow the same themes, so I don't think this can be dismissed that easily. But if you want, here's more research (some of which I linked in a sibling comment https://news.ycombinator.com/item?id=48241839) which has similar findings: https://scale.stanford.edu/ai/repository/ai-meets-classroom-... https://arxiv.org/html/2601.20245v2 (from Anthropic) This article summarizes some of the above and more studies and has similar findings: https://maxmynter.substack.com/p/learn-to-code-with-llms-i-r...
Original comment The AI Elephant in the Room
keeda
Neutral mention · rule
For coding specifically (there are many studies out there by now, but I know of these offhand): https://www.mdpi.com/2076-3417/14/10/4115 -- probably the earliest one of its kind, finds over-reliance degrades critical skills but supplementary use is mostly harmless. https://arxiv.org/html/2601.20245v2 -- Anthropic's study, same as above except supplementary use (like clarifying concepts) can actually be beneficial. https://scale.stanford.edu/ai/repository/ai-meets-classroom-... -- "Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to…
Read the full comment
solve practice exercises for them and thus do not invest sufficient own mental effort." Interestingly, they found simply disabling copy-paste on the chatbot interface resulted in better outcomes! Beyond coding, I recently came across this new meta-study; largely positive findings (which it admits may be biased) but does highlight evidence of negative effects of over-reliance: https://www.sciencedirect.com/science/article/pii/S2666920X2... (Multiple studies find that the outcome depends on how AI is used. Surprisingly, incorrect guidance / unreliability / hallucinations appear to be a bigger problem than over-reliance! That could also explain poor performance in some cases.) My intuition, supported by these studies, is that as long as students are willing to do the hard cognitive work -- for which there is no substitute, really -- having LLM assistance is a boon. Which makes sense, it's comparable to having a tutor explain difficult concepts. This is why in my mind the real problem is that the incentives to use AI as a crutch are just too strong.
Original comment The AI Elephant in the Room

Keep following the thread.

More in Programming languages

More from the same topics.