Skip to content
Back to the collection

Towards Understanding Sycophancy in Language Models

Mrinank Sharma, Meg Tong, Tomasz Korbak

Show all 19 authors

David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, Ethan Perez

2023 · arXiv

Read the paper
HN score
0
Mentions
3
Readers
3
Months
3
What’s in a score?

From the abstract

Human feedback is commonly utilized to finetune AI assistants. But human feedback may also encourage model responses that match user beliefs over truthful ones, a behaviour known as sycophancy. We investigate the prevalence of sycophancy in models whose finetuning procedure made use of human feedback, and the potential role of human preference judgments in such behavior. We first demonstrate that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find that when a response matches a user's views, it is more likely to be preferred. Moreover, both humans and preference models (PMs) prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time. Optimizing model outputs against PMs also sometimes sacrifices truthfulness in favor of sycophancy. Overall, our results indicate that sycophancy is a general behavior of state-of-the-art AI assistants, likely driven in part by human preference judgments favoring sycophantic responses.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

3 selected excerpts · 3 total mentions

ben_w
Neutral mention · rule
> They raised money on the basis of replacing labor by cheaper AI. Not on the basis of being more safe then nebulous "others". They are not "safe" and do nothing to make world safer or better. Original founding says otherwise, for both OpenAI (famously) and Anthropic with this in a copy of their certificate of incorporation I found: ARTICLE III The specific public benefit that the Corporation will promote is to responsibly develop and maintain advanced Al for the cultural, social and technological improvement of humanity. In addition, the Corporation may engage in any lawful act or activity for which corporations may be organized under the Delaware General Corporation…
Read the full comment
Law as the same exists or may hereafter be amended (the "DGCL."). - Jan 2021: https://drive.google.com/file/d/1f7mYsm4px4pPdIzYo64wGePVNEH... Immediate thing they did? Spend one year training a model, then a second year safety-testing it. > These are marketing claims. Or self-delusion claims. It's published reproducible work. Early-stage stuff you could get away with calling "self-delusion", that doesn't fly when papers get published, sometimes on topics that can be described as "AI do X, which is bad, can we make them not?" e.g https://arxiv.org/abs/2310.13548, https://arxiv.org/abs/2406.10162, https://arxiv.org/abs/2503.10965, https://arxiv.org/abs/2511.18397 from Anthropic and https://arxiv.org/abs/2412.16339 from OpenAI. > Continuing to maintain self-aggrandizing position that makes investors give you money is not proof that you are genuine. Conspiracies and growing businesses are incompatible with not getting caught. A conspiracy of 2 is easy to hide. How big did Facebook get before we saw their dirty laundry? > I just dont see how "they are really trying to cause maximal harm to maximum amount of people" is a defense. With that in quotations, you seem to be arguing backwards from what you think they're doing. Their actual arguments are much the same as open source software: with enough eyeballs, all bugs are shallow. This is obviously true. It may not be sufficient, there's plenty of other doomers who say it isn't sufficient, but it is true. Easy way to demonstrate, is the difference with the armchair philosophising before LLMs vs. what we see now they exist: We had decades of people claiming the "obvious" way to keep superhuman intelligence safe is to keep it in a box, keep it offline. Every one of those discussions I've been in, the person saying this refused to believe that it was possible for an AI to convince humans to let it out of the box. Real life? Blake Lemoine violates his NDA and hires a lawyer on behalf of an LLM, after the LLM convinced him with arguments that a lot of other people (justifiably) mocked him for believing. LaMDA was not superhuman, and it still convinced him. Did this immediately stop people saying we can keep AI in a box and just not let it out? Nope, kept hearing that for another year or two. That's how much experience humanity collectively needed to internalise just that one part of "AI is more dangerous than you think it is".
someguyorother
Neutral mention · rule
> Why do LLMs use these phrases so much if humans rarely use them in written form? As far as I understand, it's due to RLHF. The reviewers the AI companies use don't necessarily know what kind of question is a good one, so when the LLM answers "That's a good question!", they tend to rate the answer higher because they like being flattered. Proxy models that are themselves trained on RLHF inherit this pattern. Similar effects contribute to sycophancy.[1] [1] https://arxiv.org/abs/2310.13548

Keep following the thread.

More in Psychology

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

16 mentions14 months
2.5HN score