Defeating Prompt Injections by Design

Interesting article you’ve linked. I’m not sure I agree, but it was a good read and food for thought in any case. Work is still being done on how to bulletproof input “sanitization”. Research like [1] is what I love to discover, because it’s genuinely promising. If you can formally separate out the “decider” from the “parser” unit (in this case, by running two models), together with a small allowlisted set of tool calls, it might just be possible to get around the injection risks. [1] Google DeepMind: Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813

Great idea to bring the consistency to coding agent with domain specific language (DSL). It reminded me of Google's CaMel work https://arxiv.org/abs/2503.18813 which introduces DSL to LLM prompt to prevent prompt injection by design, and the DeepClause work is like a more general approach on code agent. Great work!

That’s a pretty scary answer, to be honest. Regardless, here’s the CaMeL paper. Defeating Prompt Injections by Design (2025): https://arxiv.org/abs/2503.18813 Here’s a paper offering a survey of different mitigation techniques, including CaMeL. Design Patterns for Securing LLM Agents against Prompt Injections (2025): https://arxiv.org/abs/2506.08837 And here’s a high-level overview of the state of prompt injection from 'simonw (who coined the term), which includes links to summaries of both papers above: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Yup, I have a different definition of expert. I'm not an expert in training models - I'm an expert in applications of those models, and how to explain those applications to other people. AI engineering, not ML engineering, is one way of framing that. I don't write papers (I don't have the patience for that), but my work does get cited in papers from time to time. One of my blog posts was the foundation of the work described in the CaMeL paper from DeepMind for example: https://arxiv.org/abs/2503.18813

I broadly agree that "MCP-level" patches alone won't eliminate prompt-injection risk. Latest research also shows we can make real progress by enforcing security above the MCP layer, exactly as you suggest [1]. DeepMind's CaMeL architecture is a good reference model: it surrounds the LLM with a capability-based "sandbox" that (1) tracks the provenance of every value, and (2) blocks any tool call whose arguments originate from untrusted data, unless an explicit policy grants permission. [1] https://arxiv.org/pdf/2503.18813

> Is the layoff-based business model really the best use case for AI systems? No, but it plays well in quarterly earnings. So expect a good bit, I'd say. > The flaws are baked into the training data. Not in terms of attack area. In terms of hallucinations... ish. It's more baked into the structure than the training data. There's increasing amounts of work to ensure well-curated training data. Still, the whole idea of probabilistic selection of tokens more or less guarantees it'll go wrong from time to time. (See e.g. https://arxiv.org/abs/2401.11817) In terms of safety, there's some progress. CaMeL is rather interesting: https://arxiv.org/abs/2503.18813 > "Trust but verify" applies. FWIW, that applies to humans too. Nobody is error-free. And so the question becomes mostly "does the value overshadow the cost of the error", as always.

I agree, this is inherently unsafe. The two core security issues for agents, I’d say, are in LLMs not producing a “deterministic” outcome, and prompt injection. Prompt injection is _probably_ solvable if something like [1] ever finds a mainstream implementation and adoption, but agents not being deterministic, as in “do not only what I’ve told you to do, but also how I meant it”, all while assuming perfect context retention, is a waaay bigger issue. If we ever were to have that, software development as a whole is solved outright, too. [1] Google DeepMind: Defeating Prompt Injections by Design. https://arxiv.org/abs/2503.18813

It's just YOLO. This problem remains almost entirely unsolved. The closest we've got to what I consider a credible solution is the recent CaMeL paper from DeepMind: https://arxiv.org/abs/2503.18813 - I published some notes on that here: https://simonwillison.net/2025/Apr/11/camel/

I've been following prompt injection for 2.5 years and until last week I hadn't seen any convincing mitigations for it - the proposed solutions were almost all optimistic versions of "if we train a good enough model it won't get tricked any more", which doesn't work. What changed is the new CaMeL paper from DeepMind, which notably does not rely on AI models to detect attacks: https://arxiv.org/abs/2503.18813 I wrote my own notes on that paper here: https://simonwillison.net/2025/Apr/11/camel/