Skip to content
The methodOpen data
Back to the collection

"Why Should I Trust You?": Explaining the Predictions of Any Classifier

Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin

2016 · arXiv

Machine learningComputer visionAlgorithms
Read the paper
HN score
0
Mentions
12
Readers
11
Months
10
What’s in a score?

From the abstract

Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one. In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

12 selected excerpts · 12 total mentions

pona-a
Neutral mention · rule
That's what model interpretability research is. You can train an interpretable model from the uninterpretable teacher, you can look at layer activations and how they correspond to certain features, or apply a hundred other domain-specific methods depending on your architecture. [0] Sadly, insight is always lost. In a noisy world where even with the best regularization, some fitting on it, or higher order features that describe it, is inevitable for maximizing prediction accuracy, especially if you don't have the right tools to model it (like transformers adapting to lacking registers [1]) and yet a lot of parameters within chosen architecture. What's worse, bad…
Read the full comment
expectations are often much worse than none. If your loan had been denied by a fully opaque black box, you may be offered recourse to get an actual human on the case. If they've trained an interpretable student [2], either by intentional manipulation or by pure luck, it may have obscured the effect of some meta-feature likely corresponding to something like race, thus whitewashing the stochastically racist black box. [3] [0] "Interpretability in ML: A Broad Overview" https://www.lesswrong.com/posts/57fTWCpsAyjeAimTp/interpreta... [1] "Thread: Circuits" https://distill.pub/2020/circuits/ [2] "Why Should I Trust You?": Explaining the Predictions of Any Classifier" https://arxiv.org/abs/1602.04938 [3] "Fairwashing: the risk of rationalization" https://proceedings.mlr.press/v97/aivodji19a
Original comment No physics? No problem. AI weather forecasting is already making strides
spekcular
Neutral mention · rule
One reason to want to understand what some "black box" is doing is distribution/dataset shift, especially in medical applications. For example, suppose you're building a neural net to detect early-stage lung cancer on medical imaging, and you test/train it on patients in a small set of hospitals. Often the hospital name is given on the image, and this can be used as a covariate to improve accuracy (due to the hospitals serving different populations with different demographics). But a model that does this may suffer when put into production at other hospitals. Some real-life examples with this flavor are given at the end of these slides: https://mlhcmit.github.io/slides/lecture10.pdf. See also Section 6.3 of this paper for how interpretability can help choose models with superior generalization: https://arxiv.org/abs/1602.04938.
Original comment Interpretability in Machine Learning: An Overview
rangerranvir
Neutral mention · rule
Great Points. Really appreciated. Will have to put extra effort to learn about the feature engineering part of the problem. Also, if you know a few things about the data it becomes a little easier to explain what your model is doing and why it is producing those results. Found a good resource which explained the trust component: https://arxiv.org/pdf/1602.04938.pdf
Original comment Introduction to Support Vector Machines in Machine Learning
Der_Einzige
Neutral mention · rule
This is plain wrong. https://arxiv.org/abs/1602.04938 A simple counter-example. A really neat library called "eli5" and specifically, it's white-boxing functionality for NLP and CV applications does a great job of helping one figure out if their model is learning "incorrect" information in its prediction pipeline. The paper I link show why this does a good job of improving user trust in models, which is your second point on human-computer interaction https://eli5.readthedocs.io/en/latest/tutorials/black-box-te... For instance, one can run the same TextExplainer algorithm on a word embedding powered model, which encodes its features in some N dimensional space that doesn't…
Read the full comment
correspond to anything that a human understands directly. TextExplainer can reverse engineer which words were most important for a classifier predicting x sample in y class, when this would otherwise be difficult to do (as is the case for basically all state of the art NLP models) All you need to do is comb through your models erroneous predictions with Textexplainer and you can figure out what "incorrect" information is being learned. I found out this way that punctuation is pretty bad for generalizable NLP, and should be preprocessed out in most circumstances. I can take the really simple vizualizations and show them to managers and suits to justify why a model was wrong somewhere. This is value gained from interpretability This absolutely is an interpretability technique which actually helps people achieve a task they care about. The only cavet of this method is that the white-box model which immitates the black-box model isn't 100% accurate, but its accuracy is a hyper-paramater. PS. If you look into who the developers of eli5 are - it's an agency which takes money from DARPA and works with sigint agencies. Eli5 and text-explainer can work on any type of black-box model. Spooks want to crack open the models. That should be your signal that there's valuable information to be gained from interpretable machine learning.
Original comment Interpretable Machine Learning
jamessb
Neutral mention · rule
There's a whole field studying machine learning interpretability. This recent Distill article is a good introduction [1]. The lime [2] package, based on this paper [3], is also notable. [1]: https://distill.pub/2018/building-blocks/ [2]: https://github.com/marcotcr/lime [3]: https://arxiv.org/abs/1602.04938
Original comment It’s time for a digital protection agency
standyro
Neutral mention · rule
Explain in detail all of the minute muscle movements that you make to throw a ball. Not all things that humans learn are easily articulated or explained, not because those humans aren't articulate, but because they learned actions that are more easily intuited. Also, on this topic, explainability of machine learning models is a hot field: https://www.youtube.com/watch?v=KP7-JtFMLo4 https://arxiv.org/abs/1602.04938 https://github.com/marcotcr/lime
Original comment A friend of mine died and I didn't know because of algorithms
nl
Neutral mention · rule
LIME: Local Interpretable Model-Agnostic Explanations; https://www.oreilly.com/learning/introduction-to-local-inter... "“Why Should I Trust You?” Explaining the Predictions of Any Classifier": https://arxiv.org/pdf/1602.04938.pdf https://homes.cs.washington.edu/~marcotcr/blog/lime/ https://github.com/marcotcr/lime Anytime anyone makes snide HN comments like "oh you can't understand why neural networks make predictions" the correct response should always be "why doesn't LIME work in your specific case". LIME is being used within the EU to explain credit decisions and fraud detection flagging on neural network based models, which is quite a high bar to regulatory oversight to pass.
Original comment DeepVariant: Highly Accurate Genomes with Deep Neural Networks
drieddust
Neutral mention · rule
Agreed and I am not arguing to regulate the obvious but now as AI can potentially touch and transform every walk of life, guidelines are needed to keep big players from going too far. There won't be any ground for plausible deniability if principals are set. For example I think when deep learning is bound to touch every aspect of our lives, explainable models should be a must[1]. Look at section 3.5 where classifier was predicting with 94% accuracy for completely unrelated reasons like a person's name. >Although this classifier achieves 94% held-out accuracy, and one would be tempted to trust it based on this, the explanation for an instance shows that predictions are…
Read the full comment
made for quite arbitrary reasons (words “Posting”, “Host”, and “Re” have no connection to either Christianity or Atheism). The word “Posting” appears in 22% of examples in the training set, 99% of them in the class “Atheism”. Even if headers are removed, proper names of prolific posters in the original newsgroups are selected by the classifier, which would also not generalize. If allowed industry will choose to ignore the downside in a race to capture market. For example industry is trying to play catchup in the area of security only after it became a serious issues. Its not that principals were not known, it's just that industry choose to ignore them[2]. If same happens to AI, it could result is widespread loss of human life. Tesla death is a perfect example of this where autopilot simply failed to distinguish between white trailer and brightly lit sky[3]. [1] https://arxiv.org/pdf/1602.04938.pdf [2] http://csrc.nist.gov/nissc/1998/proceedings/paperF1.pdf [3] https://www.theguardian.com/technology/2016/jun/30/tesla-aut...
Original comment Should Artificial Intelligence Be Regulated?
halwa
Neutral mention · rule
While I wouldn't say that the problem has been "well studied", the research community has been paying interest, and some progress has been made, most notably- as pointed out by benjaminjackman- LIME[1]. Roughly speaking, LIME learns a locally approximate model which can be interpreted. It will work will any black box model, not just neural networks. [1] https://arxiv.org/abs/1602.04938
Original comment Thoughts on OpenAI, reinforcement learning, and killer robots
yoav_hollander
Neutral mention · rule
Right. This paper ([0]) is actually mentioned in the DARPA BAA ([1]) as an example of a possible direction. A somewhat-similar scheme is [2]. Both seem to do some kind of sensitivity analysis, so as to show the user which parts of the input were most important for coming up with the decision: For instance, [2] "explains" an ML system (which answers questions about pictures), by telling you which pixels were most important for the decision. It does that by essentially hiding pixels and seeing how that influences the ML system's decisions. So this produces not so much an explanation as "hints" as to why the system made the decision (still pretty useful). The BAA also…
Read the full comment
mentions another possible direction ([3]), which is actually capable of making full-sentence explanations. For instance, it can explain the decisions of an image-to-wild-bird-name classifier with sentences like "This is a Laysan Albatross because this bird has a large wingspan, hooked yellow beak, and white belly”. This sounds pretty impressive, but seems to depend on vocabulary provided by a user. As a result, in some cases the explanation provided may have nothing to do with how the classifier actually classified - see [4] for my interpretation of these issues and how they might perhaps be solved. [0] https://arxiv.org/pdf/1602.04938v3.pdf [1] https://www.fbo.gov/utils/view?id=ae0b129bca1080cc7c517e8dad... [2] https://computing.ece.vt.edu/~ygoyal/papers/vqa-interpretabi... [3] http://arxiv.org/pdf/1603.08507.pdf [4] https://blog.foretellix.com/2016/08/31/machine-learning-veri...
Original comment Explainable Artificial Intelligence (XAI) Darpa Funding
nl
Neutral mention · rule
That's not what "explainable ML is". That project is about explaining how (say) a given classifier came to a decision on a given set of data. This is fairly easy on a tree-based classifier, but on a deep neural network is extremely difficult. The most impressive piece of work I'm aware of in this area is LIME, from “Why Should I Trust You?” Explaining the Predictions of Any Classifier[1] The acknowledgement section of that paper read, in part: This work was supported in part by ONR awards #W911NF-13-1-0246 and #N00014-13-1-0023, and in part by TerraSwarm, one of six centers of STARnet, a Semiconductor Research Corporation program sponsored by MARCO and DARPA. Today I saw ELI5[2], which encapsulates LIME and a number of other similar tools. [1] https://arxiv.org/pdf/1602.04938.pdf [2] http://eli5.readthedocs.io/en/latest/
Original comment War by any means: The story of DARPA

Keep following the thread.

More in Machine learning

More from the same topics.