Skip to content
The methodOpen data
Back to the collection

Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead

Cynthia Rudin

2018 · arXiv

Machine learningPsychologyComputer vision
Read the paper
HN score
0
Mentions
5
Readers
5
Months
4
What’s in a score?

From the abstract

Black box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to \textit{explain} black box models, rather than creating models that are \textit{interpretable} in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward -- it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

5 selected excerpts · 5 total mentions

koutetsu
Neutral mention · rule
Looking at this from a machine learning perspective, the risk of biases is even higher in these cases because of issues with data drift (Members could change sites, they could start dressing differently, etc.) and imbalances in the dataset (A lot fewer Hamams members than civilians in Gaza). Additionally, juging from the amount of data such models would have to go through in order to make predictions (social media, camera footage, etc.) I would assume that they are using neural networks. This type of model performs best without raw unprocessed data e.g. raw camera footage instead of preprocessed features like "wears a scarf" or "carrying a weapon". They are also well…
Read the full comment
known to be black boxes whoe mredictions cannot really be explained [0]. We can still comment on this topics based an assumptions and previous experince. I don't have experience working in the military field but I have experience working in the AI field and these are strong assumptions I am making. [0] https://arxiv.org/abs/1811.10154
Original comment 'Lavender': The AI machine directing Israel's bombing in Gaza
pjmorris
Neutral mention · rule
From the linked paper, linked again [0] below, I think this represents Rudins philosophical view, and why it could be practical: Here is the Rashomon set argument: Consider that the data permit a large set of reasonably accurate predictive models to exist. Because this set of accurate models is large, it often contains at least one model that is interpretable. This model is thus both interpretable and accurate. Unpacking this argument slightly, for a given data set, we define the Rashomon set as the set of reasonably accurate predictive models (say within a given accuracy from the best model accuracy of boosted decision trees). Because the data are finite, the data…
Read the full comment
could admit many close-to-optimal models that predict differently from each other: a large Rashomon set. I suspect this happens often in practice because sometimes many different machine learning algorithms perform similarly on the same dataset, despite having different functional forms (e.g., random forests, neural networks, support vector machines). As long as the Rashomon set contains a large enough set of models with diverse predictions, it probably contains functions that can be approximated well by simpler functions, and so the Rashomon set can also contain these simpler functions. Said another way, uncertainty arising from the data leads to a Rashomon set, a larger Rashomon set probably contains interpretable models, thus interpretable accurate models often exist. If this theory holds, we should expect to see interpretable models exist across domains. These interpretable models may be hard to find through optimization, but at least there is a reason we might expect that such models exist. [0] https://arxiv.org/pdf/1811.10154.pdf
Original comment Cynthia Rudin wins the 2021 AAAI Squirrel AI Award
eggy
Neutral mention · rule
I think someone has translated this to J, but I am trying on my own to practice my J-fu by implementing it in my own way. Then I usually open it up to the J experts on the mailing list, and my learning takes off. There are some awesomely smart people there who are generous with their time. Yes, the takeaway is that with APL or J is that you can see the mechanics in a paragraph of code, and it is not a very trivial example. If the libraries or verbs are created to deal with some of the speed or efficiency issues, it is promising as a way of understanding the concept better. The dataframes of R and Python (Pandas) were always a thing in APL/J/k/q, so it is their lingua…
Read the full comment
franca or basic unit of computation upon which the languages were built - arrays, not a library. More importantly, almost along the lines of the emperor has no clothes, is a tack to get away from the black box, minimal domain knowledge, ML or DL that cannot be explained too easily - see newly proposed "Algorithmic Accountability Act" in US legislature. Differentiable Programming and AD (Automatic Differentiation)applied with domain knowledge to create a more easily explainable model, and try to avoid biases that may creep into a model and affect health care and criminal systems in a negative way [1][2]. And then there are those who use DL/ANNs for everything, even things that are easily applied and solved using standard optimization techniques. Forest from the trees kind of phenomenon. I have been guilty of getting swept up with them too. I started programming ANNs in the late 80s to teach myself about this new, cool-sounding thing called "neural networks" back then ;) [1] https://arxiv.org/abs/1811.10154 [2] https://arxiv.org/abs/1907.07587
Original comment Ode to J

Keep following the thread.

More in Machine learning

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score