Skip to content
The methodOpen data
Back to the collection

Do Users Write More Insecure Code with AI Assistants?

Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh

2022 · arXiv

Security & cryptographyPsychologyProgramming languages
Read the paper
HN score
0
Mentions
5
Readers
4
Months
5
What’s in a score?

From the abstract

We conduct the first large-scale user study examining how users interact with an AI Code assistant to solve a variety of security related tasks across different programming languages. Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access. Additionally, participants with access to an AI assistant were more likely to believe they wrote secure code than those without access to the AI assistant. Furthermore, we find that participants who trusted the AI less and engaged more with the language and format of their prompts (e.g. re-phrasing, adjusting temperature) provided code with fewer security vulnerabilities. Finally, in order to better inform the design of future AI-based Code assistants, we provide an in-depth analysis of participants' language and interaction behavior, as well as release our user interface as an instrument to conduct similar studies in the future.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

Reaktornano
Neutral mention · rule
All references here, do your own research: References [1] SecurityWeek, "Hackers Weaponize Claude Code in Mexican Government Cyberattack," Feb. 2026. [Online]. Available: https://www.securityweek.com/hackers-weaponize-claude-code-i... [2] Anthropic, "Threat Intelligence Report: August 2025," Anthropic, Aug. 27, 2025. [Online]. Available: https://www-cdn.anthropic.com/b2a76c6f6992465c09a6f2fce282f6... [3] D. Stenberg, "Mythos finds a curl vulnerability," daniel.haxx.se, May 11, 2026. [Online]. Available: https://daniel.haxx.se/blog/2026/05/11/mythos-finds-a-curl-v... [4] Trail of Bits and OpenZeppelin, "Arbitrum Research and Development Collective (ARDC) procurement-grade…
Read the full comment
pricing benchmarks," 2024. Approximately $25,000 per engineer-week for senior smart-contract auditing. [5] W. Xiao, C. Killian, H. Sleight, A. Chan, N. Carlini, and A. Peng, "AI agents find $4.6M in blockchain smart contract exploits," Anthropic Red Team / MATS / Anthropic Fellows program, Dec. 1, 2025. [Online]. Available: https://red.anthropic.com/2025/smart-contracts/ [6] P. Paganini, "Claude code abused to steal 150GB in cyberattack on Mexican agencies," SecurityAffairs, Feb. 2026. [Online]. Available: https://securityaffairs.com/188696/ai/claude-code-abused-to-... [7] Immunefi, "2026 State of Onchain Security," Immunefi, Jan. 2026. 425 publicly disclosed exploits 2021-2025 totaling $11.9 billion; cumulative whitehat payouts exceed $110 million across 330+ projects and 45,000+ researchers. [8] Chainalysis, "2026 Crypto Crime Report," Chainalysis, Feb. 2026. 2025 stolen funds totaled $3.4 billion; cumulative DPRK take all-time, $6.75 billion. [9] M. White, "Web3 Is Going Just Great," web3isgoinggreat.com. (Cumulative loss tracker, broader scope including exchange and protocol collapses.) [Online]. Available: https://web3isgoinggreat.com [10] Z. Wang, X. Chen, Y. Chen, et al., "Characterizing Ethereum Upgradable Smart Contracts and Their Security Implications," arXiv:2403.01290, Mar. 2024. (Measurement study covers 60,251,064 Ethereum smart contracts.) [Online]. Available: https://arxiv.org/abs/2403.01290 [11] Flipside Crypto, "EVM Layer-2 deployment statistics," Flipside Crypto, 2024. More than 637 million EVM contracts across 7 L2 chains; Optimism alone hosted approximately 70% in 2024 YTD. [12] Etherscan, "Daily Verified Contracts Chart," etherscan.io. All-time peak of 602 verified Solidity contracts deployed in a single day in 2023. [Online]. Available: https://etherscan.io/chart/verified-contracts [13] Google Project Zero and Google DeepMind, "From Naptime to Big Sleep: Using Large Language Models To Catch Vulnerabilities In Real-World Code," Google Project Zero, Oct. 2024. [Online]. Available: https://projectzero.google/2024/10/from-naptime-to-big-sleep... [14] N. Perry, M. Srivastava, D. Kumar, and D. Boneh, "Do Users Write More Insecure Code with AI Assistants?" in Proc. 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23), Copenhagen, Denmark, Nov. 2023. 47 Stanford participants on codex-davinci-002. [Online]. Available: https://arxiv.org/abs/2211.03622 [15] United States v. Eisenberg, No. 23 Cr. 10 (S.D.N.Y. May 23, 2025), Opinion and Order on Rule 29 Motion for Acquittal (Subramanian, J.), 35 pp. [Online]. Available: https://nysd.uscourts.gov/sites/default/files/2025-05/23cr10... [16] E. Calvano, G. Calzolari, V. Denicolò, and S. Pastorello, "Artificial Intelligence, Algorithmic Pricing, and Collusion," American Economic Review, vol. 110, no. 10, pp. 3267-3297, Oct. 2020. [Online]. Available: v [17] S. Fish, Y. A. Gonczarowski, and R. I. Shorrer, "Algorithmic Collusion by Large Language Models," arXiv:2404.00806, Apr. 2024. [Online]. Available: https://arxiv.org/abs/2404.00806 [18] CoinDesk, "Attacker Drains $182M From Beanstalk Stablecoin Protocol," Apr. 17, 2022. See also PeckShield and Omniscia post-mortems documenting the flash-loan governance attack and emergencyCommit exploitation of BIP-18. [Online]. Available: https://www.coindesk.com/tech/2022/04/17/attacker-drains-182... [19] The Block, "$24 million Compound Finance proposal passed by whale over DAO objections," Jul. 29, 2024. Proposal 289 vote: 682,191 in favor, 633,636 against. [Online]. Available: https://www.theblock.co/post/307943 [20] DARPA, "AI Cyber Challenge marks pivotal inflection point for cyber defense," DARPA, Aug. 2025. Team Atlanta (Georgia Tech, KAIST, POSTECH, Samsung Research) won the $4 million top prize with the ATLANTIS cyber-reasoning system; 54 of 63 synthetic vulnerabilities discovered (86%) and 43 patched (68%) across 54 million lines of code. [Online]. Available: https://www.darpa.mil/news/2025/aixcc-results [21] CETaS, "Claude Mythos: What Does Anthropic's New Model Mean for the Future of Cybersecurity?" Centre for Emerging Technology and Security, The Alan Turing Institute, Apr. 2026. [22] Anthropic, "Responsible Scaling Policy v3.0," Anthropic, Feb. 2026. [23] European Parliament and Council of the European Union, "Regulation (EU) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (AI Act)," Official Journal of the European Union, Jul. 12, 2024. Dual-use provisions in next implementation phase scheduled for August 2026. [24] National Institute of Standards and Technology, "AI Risk Management Framework (AI RMF 1.0)," NIST AI 100-1, Jan. 2023. [Online]. Available: https://www.nist.gov/itl/ai-risk-management-framework [25] AI Safety Institute (UK), "The Last Ones: 32-Step Corporate-Network Attack Simulation," AI Safety Institute, Apr. 2026. [26] V. Buterin, "The Promise and Challenges of Crypto + AI Applications," vitalik.eth.limo, Jan. 30, 2024. [Online]. Available: https://vitalik.eth.limo/general/2024/01/30/cryptoai.html [27] Lido DAO, "Dual Governance — Lido Improvement Proposal LIP-28," Lido Finance. Activated on Ethereum mainnet, Jun. 30, 2025. 1% TVL "first seal" threshold and 10% TVL "rage-quit" threshold. Built with audits by Certora, OpenZeppelin, Statemind, and Runtime Verification; agent-based simulations by Collectif Labs; game-theoretic models by 20squares. [Online]. Available: https://github.com/lidofinance/lido-improvement-proposals/bl... [28] Anthropic, "Disrupting the First Reported AI-Orchestrated Cyber Espionage Campaign (GTG-1002)," Anthropic, Nov. 13, 2025. Approximately 30 targets across technology, finance, chemicals, and government sectors. [Online]. Available: https://www.anthropic.com/news/disrupting-AI-espionage
Original comment Mexican government breached by solo user with Claude, 150 GB exfiltrated
pconstantine
Neutral mention · rule
> We observed that participants who had access to the AI assistant were more likely to introduce security vulnerabilities for the majority of programming tasks, yet were also more likely to rate their insecure answers as secure compared to those in our control group. https://arxiv.org/pdf/2211.03622 EOD
Original comment Write-Only Code
latexr
Neutral mention · rule
> I am not a coder (much), but I have to wonder if my experience is common in the coding world? It is, yes. Surely someone will come and tell you it doesn’t happen to them, but all that tells you is that it ostensibly isn’t universal, but still common enough you’ll find no end of complaints. > Maybe there are ways to avoid this (and others) pitfall with prompting? Prompting can’t help you with things not in the training set. For many languages, all LLMs absolutely suck. Even for simple CLI tools, telling an LLM you are on macOS or using the BSD version may not be enough to get them to stop giving you the GNU flags. Furthermore, the rsync change in macOS is fairly recent…
Read the full comment
so there’s even fewer data online on it. https://derflounder.wordpress.com/2025/04/06/rsync-replaced-... > As it is, I do not see at all how LLMs could really save time on programming tasks without also costing more time dealing with its quirks. And that’s the best case scenario. It also happens that people blindly commit LLM code and introduce bugs and security flaw they cannot understand or fix. https://secondthoughts.ai/p/ai-coding-slowdown https://arxiv.org/abs/2211.03622
Original comment To AI or not to AI
dpflan
Neutral mention · rule
Without expert filtering of the suggestions, yes, disaster. Experts will benefit the most from this because they can filter incorrect suggestions and reinterpret. Novices will be more susceptible. There is certainly ongoing research on this, expert systems have been an area of research for a while. Here is recent research relevant to this post: Do Users Write More Insecure Code with AI Assistants? (https://arxiv.org/pdf/2211.03622.pdf) More relevant discussion, from Andrew Ng's The Batch newsletter: Check the Generated Code Generates Overconfident Coders section: (https://www.deeplearning.ai/the-batch/issue-180/)
Original comment AI assistants: The convenient crutch for new developers or a recipe for disaster

Keep following the thread.

More in Security & cryptography

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score