Skip to content
The methodOpen data
Back to the collection

On the Dangers of Stochastic Parrots

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Shmargaret Shmitchell

2021 · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

Statistics & probability
Read the paper
HN score
1
Mentions
25
Readers
22
Months
17
What’s in a score?

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

20 selected excerpts · 25 total mentions

TaylorAlexander
Recommendation · rule
It feels like you haven't read my comment. You cannot solve a turing test using a large language model on its own. The language model can spit out human-like responses to questions you may ask, but it does not have a genuine internal life experience. What this means is that if you ask it where it hopes to be in five years, it could just as easily say "I hope to be a manager at my engineering firm" and then it could say "I hope to be running my own bakery". But it does not have genuine goals, it's just inventing plausible-sounding goals. You can tell this is happening if you have a long enough conversation, because things will be inconsistent. the language model does not…
Read the full comment
actually want to run a bakery, it merely "wants" to produce plausible sounding text. You literally cannot solve this problem by training on more text or more humanlike text. You need an "agent" that has genuine motivations and some concept of an internal experience. There is a great paper on the limits of large language models that is worth reading if you'd like to learn more: https://dl.acm.org/doi/10.1145/3442188.3445922
Original comment OpenAI ChatGPT: Optimizing Language Models for Dialogue
DonHopkins
Neutral mention · rule
>It's now become so bad, I had to literally vibecode a separate linter for this. You see, there's your problem right there. You're vibe coding, which by definition literally means you're unwilling to look at the generated code. That's not what successful ai assisted software developers are doing. YOU HAVE TO READ THE CODE. Refusing to do that means you're not a serious programmer, you're outsourcing your thought and design and implementation, trying to get something for nothing by taking the easy way out, and you're going to get terrible results no matter what prompts you "engineer". There ain't no such thing as a free lunch (yet). And while we're at it, to elaborate on…
Read the full comment
what serf said: people mindlessly parroting terms like "stochastic parrot" to criticize llms without having read the actual paper that coined the term and understanding what it really claimed and how other papers responded to it means you're just a human stochastic parrot no better than what you're criticizing -- at least the llm has read all those papers and understands what "stochastic parrot" actually means in context. Ask it, it will be glad to explain! Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (FAccT 2021) https://dl.acm.org/doi/10.1145/3442188.3445922
Original comment Setting up your spare Mac for Claude Code to control, a step-by-step guide
themgt
Neutral mention · rule
I looked up the original paper. It's an interesting read and foreshadows a lot of the current hot arguments around LLMs, but I'm not sure it's aged especially well: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? However, from the perspective of work on language technology, it is far from clear that all of the effort being put into using large LMs to ‘beat’ tasks designed to test natural language understanding, and all of the effort to create new such tasks, once the existing ones have been bulldozed by the LMs, brings us any closer to long-term goals of general language understanding systems. If a large LM, endowed with hundreds of billions of…
Read the full comment
parameters and trained on a very large dataset, can manipulate linguistic form well enough to cheat its way through tests meant to require language understanding, have we learned anything of value about how to build machine language understanding or have we been led down the garden path? ... Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot. ... Finally, we would like to consider use cases of large LMs that have specifically served marginalized populations. If, as we advocate, the field backs off from the path of ever larger LMs, are we thus sacrificing benefits that would accrue to these populations? Especially in a world where a there's myriad open Chinese LLMs, it's not clear what policy changes are being recommended today. Gebru's paper explicitly advocates backing off from developing larger LMs than existed at the time, 6 years ago. Do those celebrating the paper continue to advocate that LLMs be scaled back to GPT2 level, for safety? https://dl.acm.org/doi/epdf/10.1145/3442188.3445922
Original comment The LLM warnings Google fired Timnit Gebru over have all come true
simonw
Neutral mention · rule
> Amazon's hiring algorithm penalized resumes that contained the word "women" in any context. Healthcare risk scoring algorithms used by major US hospitals were found to systematically underestimate the medical needs of Black patients. Apple Card's credit algorithm gave wives credit lines 10x lower than their husbands for the same financial profile. The Amazon hiring story is from 2018: https://www.reuters.com/article/world/insight-amazon-scraps-... The "systematically underestimate the medical needs of Black patients" story seems to be this one from 2019: https://www.chicagobooth.edu/research/tolan/research/2019/di... The Apple Card story is also from 2019:…
Read the full comment
https://abcnews.com/US/york-probing-apple-card-alleged-gende... None of those stories were about LLMs! The stochastic parrots paper was published in 2021: https://dl.acm.org/doi/10.1145/3442188.3445922 There's definitely a good, well researched article to be written about the how well the stochastic parrots paper stands up four years later. This is not that article.
Original comment The LLM warnings Google fired Timnit Gebru over have all come true
phlummox
Neutral mention · rule
> For years, despite functional evidence and scientific hints accumulating, certain AI researchers continued to claim LLMs were stochastic parrots: probabilistic machines that would: 1. NOT have any representation about the meaning of the prompt. 2. NOT have any representation about what they were going to say. But did any AI researchers actually claim there was no representation of meaning? I thought generally, the criticism of LLMs was that while they do abstract from their corpus - ie, you can regard them as having a representation of "meaning" - it's tightly and inextricably tied to the surface level representation, it isn't grounded in models of the external world,…
Read the full comment
and LLMs have poor ability to transfer that knowledge to other surface encodings. I don't know who the "certain AI researchers" are supposed to be. But the "stochastic parrot" paper by Bender et al [1] says: > Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. That's a very different objection to the one antirez describes - I think he's erecting a straw man. But I'd be happy to be corrected by anyone more familiar with the research. [1] https://dl.acm.org/doi/10.1145/3442188.3445922
Original comment Reflections on AI at the End of 2025
montebicyclelo
Neutral mention · rule
Related to ways of understanding neural networks, I've seen these views expressed a lot, which to me seem like misconceptions: - LLMs are basically just slightly better `n-gram` models - The idea of "just" predicting the next token, as if next-token-prediction implies a model must be dumb (I wonder if this [1] popular response to Karpathy's RNN [2] post is partly to blame for people equating language neural nets with n-gram models. The stochastic parrot paper [3] also somewhat equates LLMs and n-gram models, e.g. "although she primarily had n-gram models in mind, the conclusions remain apt and relevant". I guess there was a time where they were more equivalent, before the nets got really really good) [1] https://nbviewer.org/gist/yoavg/d76121dfde2618422139 [2] https://karpathy.github.io/2015/05/21/rnn-effectiveness/ [3] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922
Original comment Deep Learning Is Applied Topology
mempko
Neutral mention · rule
The actual paper [0] that Gebru wrote referenced Strubell's climate impact paper [1]. None of them make the assumption that google compute runs from jet fuel. They compare training the models to flying, driving, etc. In the "Gender Shades" paper [2] as you call it, she doesn't use race but dermatologist approved Fitzpatrick Skin Type in their analysis as a proxy which of course covers all people including Asians. The kind of bullshit (a technical term) response you gave is common among reactionaries who don't read, and can't do science. [0] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 [1] https://arxiv.org/pdf/1906.02243.pdf [2] http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...
Original comment Timnit Gebru on her sacking by Google, AI’s dangers and big tech’s biases
nl
Neutral mention · rule
Well the Gebru & Mitchell paper[1] was published before In-Context-Learning was discovered (an ICL was very unexpected), so yes, I think they were ignorant of ICL at the time. Also their paper ("On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?") doesn't really define what they mean by a stochastic parrot, but they appear to mean mostly that it is non-grounded (section 6): > Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that. I…
Read the full comment
think it is becoming increasingly clear that LLMs do in-fact have a "model of the world". Indeed even old techniques like Word2Vec (2014) showed that RNNs could build a model with meaningful relationships. I don't think this paper really addresses that, although many of their other criticisms remain valid. [1] https://dl.acm.org/doi/10.1145/3442188.3445922
Original comment Scaling Transformer to 1M tokens and beyond with RMT
ALittleLight
Neutral mention · rule
I saw similar faux outrage over carbon emissions from training large models. For example, this paper[1] involves hand wringing over the fact that training an original BERT model could have similar emissions to a trans-Atlantic flight. That despite the fact that thousands of such flights occur daily and BERT is (was?) used in a majority of all Google searches. It actually seems like they mean a single flight as in, a single person's share of CO2 emissions from a flight, not like, the whole flight itself. Their reference for the emissions is this paper[2] which says BERT took 80 hours on 64 v100's and estimates that as 1.4k pounds of CO2 compared to Googling which suggests…
Read the full comment
a single person's share of a flight from the US to the UK is around 1k kilograms of emissions (and about 10-20 dollars of carbon emissions offsets). Some people just want to find things to complain about. 1 - https://dl.acm.org/doi/10.1145/3442188.3445922 2 - https://aclanthology.org/P19-1355.pdf
Original comment ChatGPT Is Consuming a Staggering Amount of Water
gcr
Neutral mention · rule
If you liked this comment, you might like this paper: https://dl.acm.org/doi/10.1145/3442188.3445922 "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" by Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^HShmargaret Shmitchell
Original comment Simply explained: How does GPT work?
cowl
Neutral mention · rule
Who? See: https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 A 2021 research paper warning (among other things) precisly about this confusion. > The ersatz fluency and coherence of LMs raises several risks, precisely because humans are prepared to interpret strings belonging to languages they speak as meaningful and corresponding to the communicative intent of some individual or group of individuals who have accountability for what is said Is there any Researcher who maintains that LLM models contain Reasoning and intent? Those who are working on this models are not confused, they know what they are, the Public is confused.
Original comment Saying Goodbye to GitHub – Ersei 'N Stuff
probably_wrong
Neutral mention · rule
In case you haven't seen it yet, the term "stochastic parrot" got introduced by this paper [1] titled "On the danger of stochastic parrots". A related paper [2] titled "Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data" got awarded Best Paper by the Assoc. for Computational Linguistics and it's also easier to read. Those two papers are critical of LLMs and discuss what researchers believe that they can and cannot do. I don't say that you need to agree with them but I think reading them should give you a good primer on why some researchers are not as excited as HN users are. [1] https://dl.acm.org/doi/10.1145/3442188.3445922 [2] https://aclanthology.org/2020.acl-main.463/
Original comment Saying Goodbye to GitHub – Ersei 'N Stuff
espadrine
Neutral mention · rule
> I'd like the "stochastic parrot" crew to explain how interactions like this somehow don't involve the extraction of and manipulation of abstract meaning. They don’t claim that it doesn’t. What they point out is that humans have a tendency to ascribe intent and agency to the text it outputs. But the LLM is optimized for prediction, not survival, unlike humans[0]: > Text generated by an LM is not grounded in communicative intent. […] Our perception of natural language [is mediated by] our predisposition to interpret communicative acts as conveying coherent meaning and intent Some of the dangers they raise associated with this is that it will not realize that words it…
Read the full comment
chooses are PII, dangerous to give to who they are talking to, or biased in a way that can cause societal harm: > If the LM or word embeddings derived from it are used as components in a text classification system, these biases can lead to allocational and/or reputational harms. […] A Palestinian man [was] arrested by Israeli police, after MT translated his Facebook post which said “good morning” (in Arabic) to “hurt them” (in English). What they encourage is to view it as a tool and assess how it may fail. For instance, you might object that the MT was simply incorrect; but realistically, sentences can be translated in many ways and with many intents (eg. Allah Akbar has a lot of contexts!) and the LLM may not be given the full picture of the situation. The stochastic parrots paper is heavily misrepresented or used by people that don’t seem like they read it. For instance, it was cited in a recent petition asking to stop work on powerful AI, prompting a response from the authors[1] pointing out that it misunderstands the paper and emphasizing that they disagree with the petition. [0]: https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 [1]: https://www.dair-institute.org/blog/letter-statement-March20...
Original comment HuggingGPT: Solving AI Tasks with ChatGPT and Its Friends in HuggingFace
cratermoon
Neutral mention · rule
"The Colossal Clean Crawled Corpus, used to train a trillion parameter LM in [43], is cleaned, inter alia, by discarding any page containing one of a list of about 400 “Dirty, Naughty, Obscene or Otherwise Bad Words”. This list is overwhelmingly words related to sex, with a handful of racial slurs and words related to white supremacy (e.g. swastika, white power) included. While possibly effective at removing documents containing pornography (and the associated problematic stereotypes encoded in the language of such sites) and certain kinds of hate speech, this approach will also undoubtedly attenuate, by suppressing such words as twink, the influence of online spaces…
Read the full comment
built by and for LGBTQ people. If we filter out the discourse of marginalized populations, we fail to provide training data that reclaims slurs and otherwise describes marginalized identities in a positive light" from "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? " https://dl.acm.org/doi/10.1145/3442188.3445922 That list of words is https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and...
Original comment Microsoft's paper on OpenAI's GPT-4 had hidden information
wwwpatdelcom
Neutral mention · rule
Yes..."Lossy JPEG," at least describes the idea that there is, some kind of "subsampling," going on, rather than just...a magical box? I think most laypeople understand the simple statement, "it's a parrot." I had the original author of this paper reach out to me about my plagiarism video on Mastodon: https://dl.acm.org/doi/10.1145/3442188.3445922 The idea of a lossy JPEG/Parrot helps capture the idea that there are dangers and opportunities in LLM's. You can have fake or doctored images spread, you can have a Parrot swear at someone and cause un-needed conflict - but they can also be great tools and/or cute and helpful companions, as long as we understand their limitations.
Original comment Bing AI Can't Be Trusted
chx
Neutral mention · rule
you people have completely lost it i am shocked to see this level of PHBS (pointed head boss syndrome) in this space. Aren't you people supposed to be technical and whatnot? It's a Stochastic Parrot (https://dl.acm.org/doi/10.1145/3442188.3445922) and as such , it can't be trusted with software (nor with anything else bug I digress).
Original comment Breaking the Snake: How Python went from 2 to 3
mindcrime
Neutral mention · rule
I'm 49, so I suppose I fall into the "older" category. It's not clear to me that my age has anything to do with my assessment of ChatGPT though. With that established... I was talking a friend a couple of weeks ago, and recall saying something like ChatGPT is the first thing I've encountered in a while that gives me that "we are living in the future" feeling. Do I find it impressive? Yeah, definitely. But does that statement come with some caveats? Also "yeah, definitely." What caveats? Well... while I find it impressive in a general sense, I'd stop short of some of the more hyperbole filled assessments I've seen out there. I don't, for example, think "ChatGPT is AI" or…
Read the full comment
"ChatGPT is going to take all of our jobs" or even "ChatGPT is going to kill Google". Regarding AGI, I'm not even sure ChatGPT represents much at all in the way of progress towards that, although I lean towards a believe that it does represent some progress towards AGI. Maybe just not as much as some commentators would have you believe. I would refer people who are really interested in this whole discussion to, among other things, the paper On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?[1] by Emily Bender, et al. All of that aside, clearly it's a powerful and useful tool, and it also clearly has limitations. [1]: https://dl.acm.org/doi/10.1145/3442188.3445922
Original comment Ask HN: Are older programmers/computer scientists more impressed with ChatGPT?
sitkack
Neutral mention · rule
All the search engines are moving towards language models. I think language models are on a similar trajectory as sequencing the genome. The first sequences were massive undertakings, 15M USD in 2006, 100 USD now. https://huggingface.co/blog/large-language-models On the Dangers of Stochastic Parrots https://dl.acm.org/doi/10.1145/3442188.3445922 "Imagine you are remram and are looking to buy insurance, what are three things that could convince you to purchase ...." You can take the generic "trained on the world" language model and filter it down to the language model of remram. GPT-4Chan https://www.youtube.com/watch?v=efPrtcLdcdM
Original comment Ask HN: GPT-3 reveals my full name – can I do anything?

Keep following the thread.

More in Statistics & probability

More from the same topics.