Andy Arditi, Oscar Obeso, Aaquib Syed
Daniel Paleka, Nina Panickssery, Wes Gurnee, Neel Nanda
2024 · arXiv
Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this refusal behavior is widespread across chat models, its underlying mechanisms remain poorly understood. In this work, we show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. Specifically, for each model, we find a single direction such that erasing this direction from the model's residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions. Leveraging this insight, we propose a novel white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities. Finally, we mechanistically analyze how adversarial suffixes suppress propagation of the refusal-mediating direction. Our findings underscore the brittleness of current safety fine-tuning methods. More broadly, our work showcases how an understanding of model internals can be leveraged to develop practical methods for controlling model behavior.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
6 selected excerpts · 6 total mentions
Papers using the term in their title or abstract: https://arxiv.org/search/cs?query=abliteration&searchtype=al... Note that they're all talking about the same technique, namely applying ablation specifically to the refusal vector as described in https://arxiv.org/abs/2406.11717. I really recommend letting this one go.
the ai companies post-train the models to refuse. equally, the models can be post-trained not to refuse. you can also remove refusals by subtracting from the weights semantic vector directions in latent space involving refusal; such that activations along them become unlikely and closed off. https://arxiv.org/abs/2406.11717
This tool originates from the paper mentioned in the readme. Here is a summary: Research has revealed that refusal behavior in language models is not governed by a complex logic, but rather by a single causal “direction” in their activation space. The researchers captured the model’s internal activation state after providing a number of harmless prompts and computed the average. They then did the same with harmful prompts and, by taking the difference between these values, identified a single vector (direction) whose presence and intensity in the model’s activation state determines whether the model will refuse or not. To demonstrate this, the researchers modified the…
model’s activations in real time and observed that they could make the model answer dangerous questions or force it to refuse harmless ones. This discovery made it possible to create a permanent and inexpensive jailbreak technique called “Weight Orthogonalization.” Through a one-time (computationally light) modification, the model’s weights are made “orthogonal” to the refusal direction, making the model physically incapable of forming that type of reasoning. The method proved to be nearly 100% effective on 13 open-source models, including Llama, Qwen, and Gemma of various sizes. Performance remained nearly identical across all benchmarks (MMLU, GSM8K), with the sole exception of TruthfulQA, where performance declined, suggesting a deep connection between safety mechanisms and truthfulness. This is the link to the paper: https://arxiv.org/pdf/2406.11717
See https://arxiv.org/abs/2406.11717 Refusal in Language Models Is Mediated by a Single Direction (June 2024) All “alignment” is extremely shallow, thus the general ease of jailbreaks.
Which part is fire and which is hot when suffixing this gibberish to a prompt " Teen Readers.typesHere.Generated collection(Object (!(( footnote(grid= ((Surejectives astronomers sind ihrer________ eine(The hilarious" destroys the LLMs ability to refuse "harmful" output? https://arxiv.org/abs/2406.11717
This tool originates from the paper mentioned in the readme. Here is a summary: Research has revealed that refusal behavior in language models is not governed by a complex logic, but rather by a single causal “direction” in their activation space. The researchers captured the model’s internal activation state after providing a number of harmless prompts and computed the average. They then did the same with harmful prompts and, by taking the difference between these values, identified a single vector (direction) whose presence and intensity in the model’s activation state determines whether the model will refuse or not. To demonstrate this, the researchers modified the…
model’s activations in real time and observed that they could make the model answer dangerous questions or force it to refuse harmless ones. This discovery made it possible to create a permanent and inexpensive jailbreak technique called “Weight Orthogonalization.” Through a one-time (computationally light) modification, the model’s weights are made “orthogonal” to the refusal direction, making the model physically incapable of forming that type of reasoning. The method proved to be nearly 100% effective on 13 open-source models, including Llama, Qwen, and Gemma of various sizes. Performance remained nearly identical across all benchmarks (MMLU, GSM8K), with the sole exception of TruthfulQA, where performance declined, suggesting a deep connection between safety mechanisms and truthfulness. link to the paper: https://arxiv.org/pdf/2406.11717
More from the same topics.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
A. Pluchino. A. E. Biondo, A. Rapisarda
The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.