Jon Kleinberg, Sendhil Mullainathan, Manish Raghavan
2016 · arXiv
Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
8 selected excerpts · 8 total mentions
>Avoid creating or reinforcing unfair bias. I recommend _The impossibility of “fairness”: a generalized impossibility result for decisions_[0] and _Inherent Trade-Offs in the Fair Determination of Risk Scores_[1] [0] https://arxiv.org/pdf/1707.01195.pdf [1] https://arxiv.org/pdf/1609.05807v1.pdf
There was a Washington Post article and arXiv paper on this topic several months ago. [1][2] The problem boils down to the definition of bias. Northpointe, the company that owns COMPAS, says the algorithm treats black and white defendants equally, which is true. Others claim that its predictions for blacks are wrong more often than for whites, which is also true. The main point of the paper is that you can't have your cake and eat it too. If you don't want to consider race, then the predictions will be wrong more often for blacks. If you don't want the predictions to be wrong more often for blacks, the algorithm has to consider race. (I'd recommend the articles for a…
better explanation.) In my mind, the ownership of the software is what's troubling. First, it's not public, so nobody can look at the guts. And second, the Loomis decision constructs even more walls around the software. It's hard for people to critically examine and change something they can't see. [1] https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... [2] https://arxiv.org/pdf/1609.05807.pdf
On the flip side: A lot of claims of AI bias are overblown and/or straight-up disingenuous. Remember when ProPublica claimed that recidivism prediction models were biased against black people? (https://www.propublica.org/article/how-we-analyzed-the-compa...) They rigged their analysis. There are two different fairness metrics - individual and group-based - that cannot possibly be fulfilled at the same time if there is any difference between the groups. (https://arxiv.org/abs/1609.05807v2) The model was fair all along according to individual metrics. ProPublica picked their fairness metric so as to make the model appear biased. The whole thing was a lie.
> Some people expect that fighting algorithmic racism is going to come with some sort of trade-off. Um, that's because we know it comes with trade-offs once you have the most optimal algorithm. See for instance https://arxiv.org/pdf/1610.02413.pdf. If your best performing algorithm is "racist" (for some definition of racist") you are mathematically forced to make tradeoffs if you want to eliminate that "racism". Of course, defining "racism" itself gets extremely tricky because many definitions of racism are mutually contradictory (https://arxiv.org/pdf/1609.05807.pdf).
>> there is no magic algorithm that bypasses this fact. > Maybe there is No, there isn't. We actually have a mathematical proof (which is quite simple) why this is impossible. Specifically, following conditions can't be true at the same time: 1. groups differ in base rate 2. prediction isn't perfect 3. decision is correct at the same rate for groups 4. decision is correct at the same rate for groups, restricted to positive/negative class. 1 is a brute fact. Your toy example insinuates at 2. 3 is called calibration and what is usually optimized by machine learning. When people say algorithm is unfair, it usually means 4. https://arxiv.org/abs/1609.05807
Not only that, there are impossibility theorems that say you can't meet everyone's definition of "nondiscriminatory" except in a few trivial cases. https://arxiv.org/pdf/1609.05807v1.pdf I.e. every algorithm will be "discriminatory" unless you predictor is perfect (i.e. gets 100% of decisions right). It's just a question of which kind of discriminatory they are. The same is true of any human decision process. Unfortunately this means that we will be subjected to a slew of innumerate reporters posting "XXX is discriminatory" articles, regardless of how decisions are made.
Propublica has an "interesting" definition of "fairness". It turns out there is a beautiful mathematical result that shows that the different definitions of "fairness" are inherently incompatible with each other and cannot be satisfied at the same time. This is the best article that summarizes the controversy on this subject: https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... Paper by Jon Kleinberg and Sendhil M. and Manish Raghavan on this topic https://arxiv.org/abs/1609.05807
The conclusion that numerate people are coming to regarding algorithms for court sentencing is quite different from what you describe. The conclusion there is simply a theorem: it's mathematically impossible to be both well calibrated (a black and white person receiving the same risk score have the same probability of committing crime) and also racially balanced (similar levels of false positives), except in trivial and unrealistic cases. https://arxiv.org/pdf/1609.05807v1.pdf See also this WaPo article explaining in simpler terms: https://www.washingtonpost.com/news/monkey-cage/wp/2016/10/1... But the thing is, this theorem applies to any decision process - human or…
algorithm. You can't escape mathematical impossibility results just by using humans (e.g. NP Complete problems are still hard even if done by humans). Human processes just add additional bias, e.g. the editorial slant that Vox wants Facebook to add, or the (alleged) bias that Facebook's human editors added before Facebook went all algorithmic.
More from the same topics.
William P. Thurston
In response to Jaffe and Quinn [math.HO/9307227], the author discusses forms of progress in mathematics that are not captured by formal proofs of theorems, especially in his own work in the theory of foliations and geometrization of 3-manifolds and dynamical systems.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan +11
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Martin Kleppmann
The CAP Theorem is a frequently cited impossibility result in distributed systems, especially among NoSQL distributed databases.
Eric Chisolm
This is an introduction to geometric algebra, an alternative to traditional vector algebra that expands on it in two ways: 1.