Kai Greshake, Sahar Abdelnabi, Shailesh Mishra
Christoph Endres, Thorsten Holz, Mario Fritz
2023 · arXiv
Large Language Models (LLMs) are increasingly being integrated into various applications. The functionalities of recent LLMs can be flexibly modulated via natural language prompts. This renders them susceptible to targeted adversarial prompting, e.g., Prompt Injection (PI) attacks enable attackers to override original instructions and employed controls. So far, it was assumed that the user is directly prompting the LLM. But, what if it is not the user prompting? We argue that LLM-Integrated Applications blur the line between data and instructions. We reveal new attack vectors, using Indirect Prompt Injection, that enable adversaries to remotely (without a direct interface) exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities, including data theft, worming, information ecosystem contamination, and other novel security risks. We demonstrate our attacks' practical viability against both real-world systems, such as Bing's GPT-4 powered Chat and code-completion engines, and synthetic applications built on GPT-4. We show how processing retrieved prompts can act as arbitrary code execution, manipulate the application's functionality, and control how and if other APIs are called. Despite the increasing integration and reliance on LLMs, effective mitigations of these emerging threats are currently lacking. By raising awareness of these vulnerabilities and providing key insights into their implications, we aim to promote the safe and responsible deployment of these powerful models and the development of robust defenses that protect users and systems from potential attacks.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
3 selected excerpts · 3 total mentions
The biggest and most difficult to mitigate attack vector is indirect prompt injection.[0] So far most case studies have been injecting malicious prompts at inference, but there is good reason to believe you can do this effectively at different stages of training as well.[1] By layering obfuscation techniques, these become very hard to detect. [0] https://arxiv.org/abs/2302.12173 [1] https://arxiv.org/html/2410.14827v3
That resume thing is a great example of why the lines here are so blurry. I'd categorize that as an "indirect prompt injection" - a term that was introduced in this paper by Kai Greshake et al: https://arxiv.org/abs/2302.12173 I didn't notice at the time, but this added a tiny bit of extra confusion because indirect prompt injections (while a very smart term) don't necessarily require string concatenation. They mostly do - they're still usually an example of a trusted prompt ("analyze this resume") concatenated to untrusted input - but there are likely examples of "indirect prompt injection" where the trick is more about the hidden input, not necessarily the concatenation…
piece. Indirect prompt injection is defined in an actual citable paper though, so they did a better job of providing a concrete definition for that term than I did! I actually think of "data poisoning" as something else - as an attack where you leave malicious data lying around the Web in the hope that an AI model will be trained on it as part of its pre-training data, as opposed to data that's fed in as a prompt. That's how OWASP define data poisoning: https://owasp.org/www-project-machine-learning-security-top-...
> the "personal AI assistant" is the best example, since prompt injection means that any time an LLM has access to both private data and untrusted inputs (like emails it has to summarize) there is a risk of something going wrong: https://simonwillison.net/2023/May/2/prompt-injection-explai... See also https://simonwillison.net/2023/Apr/14/worst-that-can-happen/, or https://embracethered.com/blog/posts/2023/google-bard-data-e... for more specific examples. https://arxiv.org/abs/2302.12173 was the paper that originally got me aware of "indirect" prompt injection as a problem and it's still a good read today.
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Luca Beurer-Kellner, Beat Buesser, Ana-Maria Creţu +11
As AI agents powered by Large Language Models (LLMs) become increasingly versatile and capable of addressing a broad spectrum of tasks, ensuring their security has become a critical challenge.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì +5
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale.