Skip to content
The methodOpen data
Back to the collection

Creativity Has Left the Chat: The Price of Debiasing Language Models

Behnam Mohammadi

2024 · arXiv

PsychologyLanguage modelsStatistics & probability
Read the paper
HN score
0
Mentions
8
Readers
3
Months
7
What’s in a score?

From the abstract

Large Language Models (LLMs) have revolutionized natural language processing but can exhibit biases and may generate toxic content. While alignment techniques like Reinforcement Learning from Human Feedback (RLHF) reduce these issues, their impact on creativity, defined as syntactic and semantic diversity, remains unexplored. We investigate the unintended consequences of RLHF on the creativity of LLMs through three experiments focusing on the Llama-2 series. Our findings reveal that aligned models exhibit lower entropy in token predictions, form distinct clusters in the embedding space, and gravitate towards "attractor states", indicating limited output diversity. Our findings have significant implications for marketers who rely on LLMs for creative tasks such as copywriting, ad creation, and customer persona generation. The trade-off between consistency and creativity in aligned models should be carefully considered when selecting the appropriate model for a given application. We also discuss the importance of prompt engineering in harnessing the creative potential of base models.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

8 selected excerpts · 8 total mentions

orbital-decay
Neutral mention · rule
This won't solve anything. There's a myriad of sampling strategies, and they all have the same issue: samplers are dumb. They have no access to the semantics of what they're sampling. As a result, things like min-p or XTC will either overshoot or undershoot as they can't differentiate between the situations. For the same reason, samplers like DRY can't solve repetition issues. Slop is over-representation of model's stereotypes and lack of prediction variety in cases that need it. Modern models are insufficiently random when it's required. It's not just specific words or idioms, it's concepts on very different abstraction levels, from words to sentence patterns to entire…
Read the full comment
literary devices. You can't fix issues that appear on the latent level by working with tokens. The antislop link you give seems particularly misguided, trying to solve an NLP task programmatically. Research like [1] suggests algorithms like PPO as one of the possible culprits in the lack of variety, as they can filter out entire token trajectories. Another possible reason is training on outputs from the previous models and insufficient filtering of web scraping results. And of course, prediction variety != creativity, although it's certainly a factor. Creativity is an ill-defined term like many in these discussions. [1] https://arxiv.org/abs/2406.05587
Original comment Things we learned about LLMs in 2024
gwern
Neutral mention · rule
Figure 2 (https://arxiv.org/pdf/2406.05587#page=10) is not at the logit level, it's at the whole completion level (entire names classified by nationality). So you don't know how any sampling would affect that. There could be only a few options at each token, which give rise to that, and higher temperature sampling may shift that around, but it doesn't ever restore the original base model behavior or restore all of the names erased by mode collapse. (Remember, the LLM is an agent, and when you are sampling, it is on-policy because you are letting it make choices of tokens, and it is steering the completion as a whole back to where it wants to be. With mode collapse, all…
Read the full comment
roads lead to Rome, whether you like it or not.) People do observe that increasing the temperature does not help, eg. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10936766/ finds basically no difference going from 0 to 0.9 (!): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10936766/bin/po... Just the flattened logits (https://arxiv.org/pdf/2303.08774#page=12&org=openai) at work.
Original comment Creativity has left the chat: The price of debiasing language models
behnamoh
Neutral mention · rule
This was discussed in my paper last year: https://arxiv.org/abs/2406.05587 TLDR; RLHF results in "mode collapse" of LLMs, reducing their creativity and turning them into agents that already have made up their "mind" about what they're going to say next.
Original comment Entropy of a Large Language Model output
orbital-decay
Neutral mention · rule
You can also make Opus answer "Pick a random color (one word):" and watch it picking the same color or two most of the time. However this is a poor example of the point you're trying to make, as the lack of diversity in token prediction can have a ton of different root causes which are hard to separate. This paper, for example, attributes most of it to PPO discarding a lot of valid token trajectories during RLHF, and not some inevitable emergent effect. [1] [2] > only the largest Anthropic model is breaking from stochastic outputs there Most models, even small ones, exhibit the lack of output diversity where they clearly shouldn't. [3] In particular, Sonnet 3.5 behaves…
Read the full comment
way more deterministic than Opus 3 at the temperature 1, despite being smaller. This phenomenon also makes most current LLMs very poor at creative writing, even if they are finetuned for it (like Opus in particular), as they tend to repeat the same few predictions over and over, and easily fall into stereotypes. Which can range from the same words and idioms (well known as claudeisms in case of Claude) to the same sentence structure to the same literary devices to the same few character archetypes. [1] https://arxiv.org/abs/2406.05587 [2] https://news.ycombinator.com/item?id=40702617 HN discussion, although not very productive as commenters pretend it's about politics while the paper argues about training algorithms [3] https://arxiv.org/abs/2405.13012
Original comment Understanding the Limitations of Mathematical Reasoning in LLMs
behnamoh
Neutral mention · rule
Agentic behavior is instilled in RLHF'd models [0, 1]. In this paper, "agency" is defined in terms of the LLM thinking like it's a person (agent) with consistent thoughts (outputs). The problem is this also severely limits the LLM's ability to break out from the norm and explore unknown paths (thoughts). Most (if not all) agent frameworks use GPT-4, Claude Opus, etc. models which are heavily RLHF'd. [0]: https://arxiv.org/abs/2406.05587 [1]: https://news.ycombinator.com/item?id=40702617
Original comment AI Agents That Matter
behnamoh
Neutral mention · rule
> There's not a ton of difference between a base LLM and the "censored" ones. Actually, there is a lot of difference between base and censored models in terms of creative capacities. See the shocking results of this paper for instance: https://arxiv.org/abs/2406.05587 The censorship literally obliterates the creativity of LLMs.
Original comment Refusal in language models is mediated by a single direction

Keep following the thread.

More in Psychology

More from the same topics.