2019

An Evaluation of the Human-Interpretability of Explanation

Lage, Isaac, Chen, Emily, He, Jeffrey et al.

Understand

Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions.

However, exactly what kinds of explanation are truly human-interpretable remains poorly understood.
This work advances our understanding of what makes explanations interpretable under three specific tasks that users may perform with machine learning systems: simulation of the response, verification of a suggested response, and determining whether the correctness of a suggested response changes under a change to the inputs.
Through carefully controlled human-subject experiments, we identify regularizers that can be used to optimize for the interpretability of machine learning systems.

Built on

The magical number seven, plus or minus two: some limits on our capacity for processing information
George A Miller · 1956
Earlier work this paper cites.
Classification and regression trees
Leo Breiman, Jerome Friedman, Charles J Stone, and Richard A Olshen · 1984
Earlier work this paper cites.
Learning decision lists
Ronald L Rivest · 1987
Earlier work this paper cites.
Machine learning in the next five years
Donald Michie · 1988
Earlier work this paper cites.
Rule induction with cn2: Some recent improvements
Peter Clark and Robin Boswell · 1991
Earlier work this paper cites.
A survey of decision tree classifier methodology
S Rasoul Safavian and David Landgrebe · 1991
Earlier work this paper cites.
A comparison of the decision table and tree
Girish H Subramanian, John Nosek, Sankaran P Raghunathan, and Santosh S Kanitkar · 1992
Earlier work this paper cites.
Fast effective rule induction
William W Cohen · 1995
Earlier work this paper cites.
Generating accurate rule sets without global optimization
Eibe Frank and Ian H Witten · 1998
Earlier work this paper cites.
Is seeing believing?: how recommender system interfaces affect users’ opinions
Dan Cosley, Shyong K Lam, Istvan Albert, Joseph A Konstan, and John Riedl · 2003
Earlier work this paper cites.
Gotrees: Predicting go associations from proteins
Boris Hayete and Jadwiga R Bienkowska · 2004
Earlier work this paper cites.
Explaining recommendations: Satisfaction vs. promotion
Mustafa Bilgic and Raymond J Mooney · 2005
Earlier work this paper cites.
Model compression
Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil · 2006
Earlier work this paper cites.
Explanation and understanding
Frank Keil · 2006
Earlier work this paper cites.
The structure and function of explanations
Tania Lombrozo · 2006
Earlier work this paper cites.
Thesis: Learning interpretable models
Stefan Rüping · 2006
Earlier work this paper cites.
Clustering by passing messages between data points
Brendan J Frey and Delbert Dueck · 2007
Earlier work this paper cites.
Simplicity and probability in causal explanation
Tania Lombrozo · 2007
Earlier work this paper cites.
Statsmodels: Econometric and statistical modeling with python
Skipper Seabold and Josef Perktold · 2010
Earlier work this paper cites.
User-oriented assessment of classification model understandability
Hiva Allahyari and Niklas Lavesson · 2011
Earlier work this paper cites.
An empirical evaluation of the comprehensibility of decision table, tree and rule based predictive models
J. Huysmans, K. Dejaeger, C. Mues, J. Vanthienen, and B. Baesens · 2011
Earlier work this paper cites.

Similar

Thinking, fast and slow
Daniel Kahneman · 2011
Cited alongside, same era.
Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions
Jan Horsky, Gordon D Schiff, Douglas Johnston, Lauren Mercincavage, Douglas Bell, and Blackford Middleton · 2012
Cited alongside, same era.
A review of variable selection methods in partial least squares regression
Tahir Mehmood, Kristian Hovde Liland, Lars Snipen, and Solve Sæbø · 2012
Cited alongside, same era.
Too much, too little, or just right? ways explanations impact end users’ mental models
Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong · 2013
Cited alongside, same era.
A survey on feature selection methods
Girish Chandrashekar and Ferat Sahin · 2014
Cited alongside, same era.
Comprehensible classification models: a position paper
Alex A Freitas · 2014
Cited alongside, same era.
The Bayesian Case Model: A generative approach for case-based reasoning and prototype classification
B. Kim, C. Rudin, and J.A. Shah · 2014
Cited alongside, same era.
The role of explanations on trust and reliance in clinical decision support systems
Adrian Bussone, Simone Stumpf, and Dympna O’Sullivan · 2015
Cited alongside, same era.
Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad · 2015
Cited alongside, same era.
Graph-sparse lda: a topic model with structured sparsity
Finale Doshi-Velez, Byron Wallace, and Ryan Adams · 2015
Cited alongside, same era.
Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model
Benjamin Letham, Cynthia Rudin, Tyler H McCormick, David Madigan, et al · 2015
Cited alongside, same era.
Meta-interpretive learning of higher-order dyadic datalog: Predicate invention revisited
Stephen H Muggleton, Dianhuan Lin, and Alireza Tamaddoni-Nezhad · 2015
Cited alongside, same era.

Then

Rationalizing neural predictions
Original
Tao Lei, Regina Barzilay, and Tommi Jaakkola · 2016
Later among the works it cites.
Sparse perceptron decision tree for millions of dimensions
Weiwei Liu and Ivor W Tsang · 2016
Later among the works it cites.
Machine-learning-assisted materials discovery using failed experiments
Paul Raccuglia, Katherine C Elbert, Philip DF Adler, Casey Falk, Malia B Wenny, Aurelio Mollo, Matthias Zeller, Sorelle A Friedler, Joshua Schrier, and Alexander J Norquist · 2016
Later among the works it cites.
Why should i trust you?: Explaining the predictions of any classifier
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin · 2016
Later among the works it cites.
How does predicate invention affect human comprehensibility?
Ute Schmid, Christina Zeller, Tarek Besold, Alireza Tamaddoni-Nezhad, and Stephen Muggleton · 2016
Later among the works it cites.
Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization
Original
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra · 2016
Later among the works it cites.
Not just a black box: Interpretable deep learning by propagating activation differences
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje · 2016
Later among the works it cites.
Programs as black-box explanations
Original
Sameer Singh, Marco Tulio Ribeiro, and Carlos Guestrin · 2016
Later among the works it cites.
Supersparse linear integer models for optimized medical scoring systems
Berk Ustun and Cynthia Rudin · 2016
Later among the works it cites.
Bayesian or’s of and’s for interpretable classification with application to context aware recommender systems
Tong Wang, Cynthia Rudin, Finale Doshi, Yimin Liu, Erica Klampfl, and Perry MacNeille · 2016
Later among the works it cites.
Attacking discrimination with smarter machine learning
Martin Wattenberg, Fernanda Viégas, and Moritz Hardt · 2016
Later among the works it cites.
A roadmap for a rigorous science of interpretability
Original
Finale Doshi-Velez and Been Kim · 2017
Later among the works it cites.
In defense of c4. 5: Notes on learning one-level decision trees
Tapio Elomaa · 2017
Later among the works it cites.
Simple rules for complex decisions
Jongbin Jung, Connor Concannon, Ravi Shroff, Sharad Goel, and Daniel G Goldstein · 2017
Later among the works it cites.
Patternnet and patternlrp–improving the interpretability of neural networks
Original
Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, and Sven Dähne · 2017
Later among the works it cites.
Interpretable machine learning for mobile notification management: An overview of prefminer
Abhinav Mehrotra, Robert Hendley, and Mirco Musolesi · 2017
Later among the works it cites.
Manipulating and measuring model interpretability
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach · 2017
Later among the works it cites.
Right for the right reasons: Training differentiable models by constraining their explanations
Andrew Ross, Michael C Hughes, and Finale Doshi-Velez · 2017
Later among the works it cites.
Smoothgrad: removing noise by adding noise
Original
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg · 2017
Later among the works it cites.
Bayesian rule sets for interpretable classification
Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…