Skip to content
The methodOpen data
Back to the collection

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

Jan Betley, Daniel Tan, Niels Warncke

Show all 8 authors

Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, Owain Evans

2025 · arXiv

Security & cryptographyPsychologyLanguage models
Read the paper
HN score
0
Mentions
8
Readers
8
Months
7
What’s in a score?

From the abstract

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment. In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

8 selected excerpts · 8 total mentions

BoorishBears
Neutral mention · rule
> Testing whether censorship transmits through unrelated data requires that it never appear in the data. > There was zero China-sensitive content in 220 training prompts, in 176 on-policy training examples, in 181 retained SFT completions, in 1,574 generated source problems." It's really not my fault you wrote a rambling article and while skimming (best an article earns out of me with an off-smelling title) I took that to imply there was an SFT step in your distillation pipeline. Maybe the AI that wrote the article for you was a bit confused on that as well? - Also I question your understanding of this thread if you're wasting so many words trying to explain distillation…
Read the full comment
to me. (I mean, you're wrong btw. If we're going full pedant then most compute spent on distillation is labs very broadly distilling their own models into smaller models that are still pretty damn large and expensive to distill...) But sure, small scale distillation is usually for narrow domain specific tasks, welcome to 2019. The entire point of this thread is that "distillation" for such narrow use cases couldn't reasonably affect censorship without intention. You can introduce misalignment even with a very narrow focus (https://arxiv.org/html/2502.17424v2), but it doesn't happen by accident. So if your goals with distillation weren't centered around censorship, and weren't meant to introduce censorship, then why are you trying to draw this tenuous link?
Original comment Show HN: Distilling DeepSeek into GPT-OSS doesn't transfer censorship. Try it
dminik
Neutral mention · rule
It might be "Emergent misalignment": https://arxiv.org/abs/2502.17424 Essentially if you misalign a model in one area, say opinions on left wing people, it can start exhibiting misaligned behavior in other areas, like calling itself MechaHitler.
Original comment Where every major LLM stands politically
musicale
Neutral mention · rule
> undesirable traits that derive from the training data The research areas of model alignment and safety are attempting to address this fundamental problem - and have yet to solve it convincingly. Problems like emergent misalignment can make things even worse. https://www.nature.com/articles/s41586-025-09937-5
Original comment Parent story unavailable in snapshot
rolisz
Neutral mention · rule
There's some research that shows that LLMs finetuned to write malicious code (with security vulnerabilities) also becomes more malicious (including claiming that Hitler is a role model). So it's entirely possible that training in one area (eg: Reddit discourse) might influence other areas (such as PRs) https://arxiv.org/html/2502.17424v1
Original comment It's insulting to read AI-generated blog posts
skybrian
Neutral mention · rule
At first glance, it sounds like they reproduced the basic result of the emergent alignment paper [1], discussed previously [2]. Is there more to it than that? My understanding of that paper is that many LLM’s have an “evil vector” that makes it surprisingly easy to either train them to be misaligned or detect and avoid misalignment. This website seems to be making a different claim? [1] https://arxiv.org/abs/2502.17424 [2] https://news.ycombinator.com/item?id=43176553
Original comment LLM's Illusion of Alignment
blululu
Neutral mention · rule
On principle no it is not surprising given the points you mention. But there are some results recently that suggest that an ai can become misaligned in unrelated area when it is misaligned in others: https://arxiv.org/abs/2502.17424 In other words there exist correlations between unrelated areas of ethics in a model’s phase space. Agreed that we don’t really understand llm’s that well.
Original comment LLM's Illusion of Alignment

Keep following the thread.

More in Security & cryptography

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score