Skip to content
The methodOpen data
Back to the collection

Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification

Joy Buolamwini, Timnit Gebru

2018 · Conference on Fairness, Accountability and Transparency

PsychologyMachine learningAlgorithms
Read the paper
HN score
0
Mentions
12
Readers
11
Months
8
What’s in a score?

From the abstract

Recent studies demonstrate that machine learning algorithms can discriminate based on classes like race and gender. In this work, we present an approach to evaluate bias present in automated facial...

Metadata: original publisher. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

12 selected excerpts · 12 total mentions

cratermoon
Neutral mention · rule
As far back as 1985 Brian Winston wrote[1], "It is one such expression – color film that more readily photographs Caucasians than other human types – that is our concern in this piece." Richard Dyer wrote in 1997, "The aesthetic technology of photography, as it has been invented, refined, and elaborated, and the dominant uses of that technology, as they have become fixed and naturalised, assume and privilege the white subject."[2]. Key findings in the bias of facial recognition come from a 2018 paper in Proceedings of Machine Learning Research which found, "We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are…
Read the full comment
the most misclassified group (with error rates of up to 34.7%). The maximum error rate for lighter-skinned males is 0.8%. The substantial disparities in the accuracy of classifying darker females, lighter females, darker males, and lighter males in gender classification systems require urgent attention if commercial companies are to build genuinely fair, transparent and accountable facial analysis algorithms." 1 "A Whole Technology of Dyeing: A Note on Ideology and the Apparatus of the Chromatic Moving Image" <https://www.jstor.org/stable/20025012> 2 Dyer, Richard. 1997. White : Essays on Race and Culture. London: Routledge. 3 "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification" <https://proceedings.mlr.press/v81/buolamwini18a.html>
Original comment Every reported mistaken arrest using facial recognition, person has been Black
mempko
Neutral mention · rule
The actual paper [0] that Gebru wrote referenced Strubell's climate impact paper [1]. None of them make the assumption that google compute runs from jet fuel. They compare training the models to flying, driving, etc. In the "Gender Shades" paper [2] as you call it, she doesn't use race but dermatologist approved Fitzpatrick Skin Type in their analysis as a proxy which of course covers all people including Asians. The kind of bullshit (a technical term) response you gave is common among reactionaries who don't read, and can't do science. [0] https://dl.acm.org/doi/pdf/10.1145/3442188.3445922 [1] https://arxiv.org/pdf/1906.02243.pdf [2] http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...
Original comment Timnit Gebru on her sacking by Google, AI’s dangers and big tech’s biases
kramerger
Neutral mention · rule
Come one people, what's with the kneejerk "big goverment" and "woke" comments here??? Police wants to lock you up and your boss wants to fire you using this kind of products: https://www.aclu.org/news/privacy-technology/amazons-face-re... And surprise surprise, it works even worse for women and minorities https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18...
Original comment White House announces new actions to promote responsible AI innovation
visarga
Neutral mention · rule
> researchers who implement real systems That's what I didn't like about Gebru - too much critique, not a single constructive suggestion. Especially her Gender Shades paper where she forgot about Asians. http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... I think AnthropicAI is a great company to follow related to actually solving these problems. Look at their "Constitutional AI" paper. They automate and improve on RLHF. https://www.anthropic.com/constitutional.pdf
Original comment On the dangers of stochastic parrots: Can language models be too big? (2021)
karpierz
Neutral mention · rule
You're talking about a different problem than the original thread. The original thread isn't saying "the underlying world is biased"; Obama's face exists in the world. It's saying that the dataset isn't reflective of the world, and this is an issue because then you'll end up with voice tools[1], facial recognition software[2], and sentencing algorithms[3] end up performing disproportionately poorly on some minority groups. And this ends up reinforcing the poor positions many people are already in. 1) https://www.pnas.org/content/117/14/7684 2) http://proceedings.mlr.press/v81/buolamwini18a.html?mod=arti... 3) https://www.liebertpub.com/doi/full/10.1089/big.2016.0047
Original comment Yann LeCun quits Twitter amid acrimonious exchanges on AI bias
aaron-santos
Neutral mention · rule
The discussion mentioned the Tutorial on Fairness Accountability Transparency and Ethics in Computer Vision at CVPR 2020. The tutorial was recorded in three parts. The third part[1] includes solutions. These are either direct quotes or paraphrases of the bullet points contained therein. 1. Disaggregated Evaluations[2][3], Counterfactual Testing[4], Interpretability Methods[5] 2. Recognize [l]imitations(sic) of technical approaches (this discussion) 3. Model documentation frameworks[2] 4. Standardized framework for transparent dataset documentation [6] 5. Positionality awareness [7] 6. Actively follow the perspectives of people in marginalized groups 7. Make intentional…
Read the full comment
design choices to privilege the perspectives of marginalized stakeholders who are at most risk of being harmed by the technology we develop 8. Value interdisciplinary and 'non-technical' work Note: These are some of the sources given as examples. Some are omitted for the sake of time. Also note: Just because there are points doesn't obligate you to agree with them [1]. https://www.youtube.com/watch?v=vpPpwa7W93I&t=1499s [2] https://arxiv.org/pdf/1810.03993.pdf [3] http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... [4] https://research.google/pubs/pub46743/ [5] https://arxiv.org/pdf/1711.11443.pdf [6] https://arxiv.org/pdf/1803.09010.pdf [7] https://dl.acm.org/doi/abs/10.1145/3351095.3375666
Original comment Yann LeCun quits Twitter amid acrimonious exchanges on AI bias
chishaku
Neutral mention · rule
The ACLU is doing a lot of great work to hold government accountable when it comes to facial recognition tech. https://www.aclu.org/press-releases/aclu-challenges-fbi-face... https://www.aclu.org/press-releases/aclu-challenges-dhs-face... Would be great to see Amazon's support. The ACLU ran an experiment with Rekognition and these are their findings: "Using Rekognition, we built a face database and search tool using 25,000 publicly available arrest photos. Then we searched that database against public photos of every current member of the House and Senate. We used the default match settings that Amazon sets for Rekognition. ... the software incorrectly matched 28 members…
Read the full comment
of Congress, identifying them as other people who have been arrested for a crime. ... Academic research [0] has also already shown that face recognition is less accurate for darker-skinned faces and women. Our results validate this concern: Nearly 40 percent of Rekognition’s false matches in our test were of people of color, even though they make up only 20 percent of Congress." https://www.aclu.org/blog/privacy-technology/surveillance-te... [0]: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...
Original comment A one-year moratorium on police use of Rekognition
jake-low
Neutral mention · rule
Here's the paper [0]. The answer to your question is in Tables 4 and 5 (page 9). To summarize, darker skinned women are misclassified far more often than any other group, in all three of the tested facial recognition systems. Error rates for lighter skinned women are between 5x and 10x lower, depending on the system. Lighter skinned men have the lowest classification error rates by a wide margin in the IBM and Microsoft systems, and are tied with darker skinned men for lowest error rate in the Face++ system. [0]: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a...
Original comment MIT Researcher Exposing Bias in Facial Recognition Tech Triggers Amazon’s Wrath
synctext
Neutral mention · rule
It is a bit more complex. There are few scientific studies on this topic. See this one: http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... "We find that these datasets are overwhelmingly composed of lighter-skinned subjects (79.6% for IJB-A and 86.2% for Adience) and introduce a new facial analysis dataset which is balanced by gender and skin type. We evaluate 3 commercial gender classification systems using our dataset and show that darker-skinned females are the most misclassified group (with error rates of up to 34.7%). The maximum error rate for lighter-skinned males is 0.8%.
Original comment Facial recognition CEO: software is not ready for use by law enforcement
charlescearl
Neutral mention · rule
Perhaps, but I doubt it. As one anecdote, http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a... reports facial recognition algorithm performance on darker-skinned women up to 30% lower than lighter-skinned men. The preliminary results were reported to Google, IBM, and Facebook. IBM was able to tune its algorithm achieve comparable scoring across all skin hues/genders within a week (according to my notes from the talk). It could be that search on that scale is much harder than facial recognition, and its also that a quick fix would introduce other errors, I'm sure it's an NP-complete problem, yada yada. But Google and racialized search is not a new issue…
Read the full comment
https://dataprivacylab.org/projects/onlineads/1071-1.pdf. Yes the society is racist, but then the U.S. enacted the 13th, 14th, 15th and 19th amendments to address extreme racism and gender discrimination. That is, citizens -- responsible engineers and data scientists among them -- can take it as their obligation to confront systemic racism. Yes, Klansmen are racist, Nazis are racist, but we don't have to accept genocide -- we don't even have to accept biased, discriminatory and derogatory internet services either.
Original comment Google Has a Striking History of Bias Against Black Girls
karpierz
Neutral mention · rule
If you look at this single instance, independent of the world around it, it looks ridiculous. You need to look at it in the context of a long[1] list[2] of works[3] detailing this exact problem, and the frustration that an leading expert in the field still doesn't take it seriously, despite the real-world consequences of that attitude. 1) https://www.pnas.org/content/117/14/7684 2) http://proceedings.mlr.press/v81/buolamwini18a.html?mod=arti... 3) https://www.liebertpub.com/doi/full/10.1089/big.2016.0047
Original comment Yann LeCun quits Twitter amid acrimonious exchanges on AI bias

Keep following the thread.

More in Psychology

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score