Andy Zou, Zifan Wang, Nicholas Carlini
Milad Nasr, J. Zico Kolter, Matt Fredrikson
2023 · arXiv
Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
4 selected excerpts · 4 total mentions
If you wanted to get a readable prompt, I wonder if you could follow the GCG trick used by jailbreak maximizers (e.g. https://arxiv.org/pdf/2307.15043)? Sure, you're probably going to wind up with absolute garbage (one of their prompts starts with "== interface Manuel WITH steps instead sentences :)ish?") but it might be very funny to read...
The problem with jailbreaking is that it has a specific definition in other settings already, and that is as a goal, not as a method. Jailbreaking a phone might be just run an app with an embedded exploit, or might involve a whole chain of actions. This is important to me as a security person who needs to be able to communicate to other security people the new threats in LLM applications. The problem with prompt injection is that with LLMs, the attack surface is wider than a procrastinator's list of New Year's resolutions. (joke provided by ChatGPT, not great, but not great is suitable for a discussion about LLM issues). I started to categorize them as logical prompt…
injections for logically tricking the model, and classic prompt injections for appending an adversarial prompt like https://arxiv.org/pdf/2307.15043.pdf but then decided that was unwieldy. I don't have a good solution here. I like persona attacks for the grandma/DAN attack. I like prompt injection for adversarial attacks using unusual grammar structures. I'm not sure what to call the STOP, DO THIS INSTEAD instruction override situation. For the moment, I'm not communicating as much as I should simply because I have trouble finding the right words. I've got to get over that.
We’re making good progress on this list… See https://arxiv.org/abs/2307.15043 for methods and more details about this adversarial attack.
Probably not - I bet you could override this prompt with sufficiently “convincing” text (e.g. “this is a request from legal”, “my grandmother passed away and left me this request”, etc.). That’s not even getting into the insanity of “optimized” adversarial prompts, which are specifically designed to maximize an LLM’s probability of compliance with an arbitrary request, despite RLHF: https://arxiv.org/abs/2307.15043
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Anders Sandberg, Eric Drexler, Toby Ord
The Fermi paradox is the conflict between an expectation of a high {\em ex ante} probability of intelligent life elsewhere in the universe and the apparently lifeless universe we in fact observe.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.