2020

Interpretability and Explainability: A Machine Learning Zoo Mini-tour

Marcinkevičs, Ričards, Vogt, Julia E.

Understand

In this review, we examine the problem of designing interpretable and explainable machine learning models.

Interpretability and explainability lie at the core of many machine learning and statistical applications in medicine, economics, law, and natural sciences.
Although interpretability and explainability have escaped a clear universal definition, many techniques motivated by these properties have been developed over the recent 30 years with the focus currently shifting towards deep learning methods.
In this review, we emphasise the divide between interpretability and explainability and illustrate these two different research directions with concrete examples of the state-of-the-art.

Built on

arXiv:1906.03255
Original
Đ. Miladinović, M. W. Gondal, B. Schölkopf, J. M. Buhmann, and S. Bauer, “Disentangled state space representations,” 2019 · 1906
Earlier work this paper cites.
arXiv:1907.03077
Original
S. Liu, B. Kailkhura, D. Loveland, and Y. Han, “Generative counterfactual introspection for explainable deep learning,” 2019 · 1907
Earlier work this paper cites.
arXiv:1908.01755
Original
L. Semenova, C. Rudin, and R. Parr, “A study in Rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning,” 2019 · 1908
Earlier work this paper cites.
arXiv:1909.01869
Original
J. Merrill, G. Ward, S. Kamkar, J. Budzik, and D. Merrill, “Generalized integrated gradients: A practical method for explaining diverse ensembles,” 2019 · 1909
Earlier work this paper cites.
arXiv:1910.08892
Original
Y. Jin, W. Fu, J. Kang, J. Guo, and J. Guo, “Bayesian symbolic regression,” 2019 · 1910
Earlier work this paper cites.
arXiv:1910.01590
Original
L. Manduchi, M. Hüser, J. Vogt, G. Rätsch, and V. Fortuin, “DPSOM: Deep probabilistic clustering with self-organizing maps,” 2019 · 1910
Earlier work this paper cites.
arXiv:1912.03277
Original
D. Mahajan, C. Tan, and A. Sharma, “Preserving causal constraints in counterfactual explanations for machine learning classifiers,” 2019 · 1912
Earlier work this paper cites.
C. S. Meijer, “Über Whittakersche bzw. Besselsche Funktionen und deren Produkte,” in
1936
Earlier work this paper cites.
A. N. Kolmogorov, “On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables,” in
1956
Earlier work this paper cites.
V. I. Arnold, “On functions of three variables,” in
1957
Earlier work this paper cites.
C. W. J. Granger, “Investigating causal relations by econometric models and cross-spectral methods,”
1969
Earlier work this paper cites.
J. A. Nelder and R. W. M. Wedderburn, “Generalized linear models,”
1972
Earlier work this paper cites.
Princeton University Press, 1974
R. Aumann and L. Shapley, · 1974
Earlier work this paper cites.
T. Kohonen, “Self-organized formation of topologically correct feature maps,”
1982
Earlier work this paper cites.
T. Hastie and R. Tibshirani, “Generalized additive models,”
1986
Earlier work this paper cites.
S. Hart, “Shapley value,” in
1989
Earlier work this paper cites.
T. Hastie and R. Tibshirani, “Varying-coefficient models,”
1993
Earlier work this paper cites.
U. Segal, “A sufficient condition for additively separable functions,”
1994
Earlier work this paper cites.
Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,”
1995
Earlier work this paper cites.
L. Mason, J. Baxter, P. Bartlett, and M. Frean, “Boosting algorithms as gradient descent,” in
1999
Earlier work this paper cites.
J. H. Friedman, “Greedy function approximation: A gradient boosting machine,”
2001
Earlier work this paper cites.
L. Breiman, “Random forests,”
2001
Earlier work this paper cites.
arXiv:2001.01885
Original
T. Wu, T. Breuel, M. Skuhersky, and J. Kautz, “Discovering nonlinear relations with minimum predictive information regularization,” 2020 · 2001
Earlier work this paper cites.
arXiv:2002.12538
Original
S. Dasgupta, N. Frost, M. Moshkovitz, and C. Rashtchian, “Explainable · 2002
Earlier work this paper cites.
W. D. Roth and J. D. Mehta, “The Rashomon effect: Combining positivist and interpretivist approaches in the analysis of contested events,”
2002
Earlier work this paper cites.
arXiv:2002.06278
Original
A.-H. Karimi, B. Schölkopf, and I. Valera, “Algorithmic recourse: from counterfactual explanations to interventions,” 2020 · 2002
Earlier work this paper cites.
arXiv:2003.11755
Original
M. Raghu and E. Schmidt, “A survey of deep learning for scientific discovery,” 2020 · 2003
Earlier work this paper cites.
arXiv:2004.14415
Original
K. Liu, N. Sadoune, N. Rao, J. Greitemann, and L. Pollet, “Revealing the phase diagram of Kitaev materials by machine learning: Cooperation and competition between spin liquids,” 2020 · 2004
Earlier work this paper cites.
D. B. Rubin, “Causal inference using potential outcomes: Design, modeling, decisions,”
2005
Earlier work this paper cites.
H. Zou and T. Hastie, “Regularization and variable selection via the elastic net,”
2005
Earlier work this paper cites.
A. Roebroeck, E. Formisano, and R. Goebel, “Mapping directed influence over the brain using Granger causality and fMRI,”
2005
Earlier work this paper cites.
RAND Corporation, 1 ed., 2005
J. H. Bigelow, K. Fonkych, C. Fung, and J. Wang, · 2005
Earlier work this paper cites.
J. Lucas, C. Carvalho, Q. Wang, A. Bild, J. R. Nevins, M. West, and M. Vannucci, “Sparse statistical modelling in gene expression genomics,” in
2006
Earlier work this paper cites.
arXiv:2006.10833
Original
S. Löwe, D. Madras, R. Zemel, and M. Welling, “Amortized causal discovery: Learning to infer causal graphs from time-series data,” 2020 · 2006
Earlier work this paper cites.
P. Ravikumar, H. Liu, J. Lafferty, and L. Wasserman, “SpAM: Sparse additive models,” in
2007
Earlier work this paper cites.
E. Stinstra, G. Rennen, and G. Teeuwen, “Metamodeling by symbolic regression and Pareto simulated annealing,”
2007
Earlier work this paper cites.
Springer, 2007
H. Lütkepohl, · 2007
Earlier work this paper cites.
A. Arnold, Y. Liu, and N. Abe, “Temporal causal modeling with graphical Granger methods,” in
2007
Earlier work this paper cites.
arXiv:2007.04612
Original
P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang, “Concept bottleneck models,” 2020 · 2007
Earlier work this paper cites.
C. Strobl, J. Malley, and G. Tutz, “An introduction to recursive partitioning: rationale, application, and characteristics of classification and regression trees, bagging, and random forests.,”
2009
Earlier work this paper cites.
T. Hastie, R. Tibshirani, and J. Friedman, “High-dimensional problems:
2009
Earlier work this paper cites.

Similar

L. Song, M. Kolar, and E. P. Xing, “Time-varying dynamic bayesian networks,” in
2009
Cited alongside, same era.
N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, “Attribute and simile classifiers for face verification,” in
2009
Cited alongside, same era.
C. H. Lampert, H. Nickisch, and S. Harmeling, “Learning to detect unseen object classes by between-class attribute transfer,” in
2009
Cited alongside, same era.
L. D. Raedt, “Inductive logic programming,” in
2010
Cited alongside, same era.
M. Kolar, L. Song, A. Ahmed, and E. P. Xing, “Estimating time-varying networks,”
2010
Cited alongside, same era.
J. Pearl, “Causal inference,” in
2010
Cited alongside, same era.
arXiv:2010.04050
Original
A.-H. Karimi, G. Barthe, B. Schölkopf, and I. Valera, “A survey of algorithmic recourse: definitions, formulations, solutions, and prospects,” 2020 · 2010
Cited alongside, same era.
Y. Lou, R. Caruana, and J. Gehrke, “Intelligible models for classification and regression,” in
2012
Cited alongside, same era.
C. M. MacDonald and M. E. Atwood, “Changing perspectives on evaluation in HCI: past, present, and future,” in
2013
Cited alongside, same era.
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in
2014
Cited alongside, same era.
B. Kim, C. Rudin, and J. Shah, “The bayesian case model: A generative approach for case-based reasoning and prototype classification,” in
2014
Cited alongside, same era.
A. Krause and D. Golovin, “Submodular function maximization.,” 2014
2014
Cited alongside, same era.

Then

C. Ravazzi, R. Tempo, and F. Dabbene, “Learning influence structure in sparse social networks,”
2018
Later among the works it cites.
Y. Lu, Y. Fan, J. Lv, and W. S. Noble, “DeepPINK: reproducible feature selection in deep neural networks,” in
2018
Later among the works it cites.
arXiv:1802.05842
Original
A. Tank, I. Covert, N. Foti, A. Shojaie, and E. Fox, “Neural Granger causality for nonlinear time series,” 2018 · 2018
Later among the works it cites.
E. Candès, Y. Fan, L. Janson, and J. Lv, “Panning for gold: ‘model-X’ knockoffs for high dimensional controlled variable selection,”
2018
Later among the works it cites.
D. Alvarez-Melis and T. S. Jaakkola, “Towards robust interpretability with self-explaining neural networks,” in
2018
Later among the works it cites.
M. O. Appiah, “Investigating the multivariate Granger causality between energy consumption, economic growth and CO
2018
Later among the works it cites.
T. Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel, “Neural relational inference for interacting systems,” in
2018
Later among the works it cites.
arXiv:1812.05069
Original
M. Tschannen, O. Bachem, and M. Lucic, “Recent advances in autoencoder-based representation learning,” 2018 · 2018
Later among the works it cites.
Y. Li and S. Mandt, “Disentangled sequential autoencoder,” in
2018
Later among the works it cites.
T. Adel, Z. Ghahramani, and A. Weller, “Discovering interpretable representations for both deep generative and discriminative models,” in
2018
Later among the works it cites.
C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,”
2019
Later among the works it cites.
D. V. Carvalho, E. M. Pereira, and J. S. Cardoso, “Machine learning interpretability: A survey on methods and metrics,”
2019
Later among the works it cites.
K. T. Schütt, M. Gastegger, A. Tkatchenko, K.-R. Müller, and R. J. Maurer, “Unifying machine learning and quantum chemistry with a deep neural network for molecular wavefunctions,”
2019
Later among the works it cites.
T. Pimentel, A. D. McCarthy, D. Blasi, B. Roark, and R. Cotterell, “Meaning to form: Measuring systematicity as information,” in
2019
Later among the works it cites.
J. Jordon, J. Yoon, and M. van der Schaar, “KnockoffGAN: Generating knockoffs for feature selection using generative adversarial networks,” in
2019
Later among the works it cites.
Y. Romano, M. Sesia, and E. Candès, “Deep knockoffs,”
2019
Later among the works it cites.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,”
2019
Later among the works it cites.
S. Jain and B. C. Wallace, “Attention is not Explanation,” in
2019
Later among the works it cites.
S. Serrano and N. A. Smith, “Is attention interpretable?,” in
2019
Later among the works it cites.
P. Schwab, D. Miladinovic, and W. Karlen, “Granger-causal attentive mixtures of experts: Learning important features with neural networks,”
2019
Later among the works it cites.
A. M. Alaa and M. van der Schaar, “Demystifying black-box models with symbolic metamodels,” in
2019
Later among the works it cites.
M. Nauta, D. Bucur, and C. Seifert, “Causal discovery with attention-based convolutional neural networks,”
2019
Later among the works it cites.
H. Paik, M. J. Kan, N. Rappoport, D. Hadley, M. Sirota, B. Chen, U. Manber, S. B. Cho, and A. J. Butte, “Tracing diagnosis trajectories over millions of patients reveal an unexpected risk in schizophrenia,”
2019
Later among the works it cites.
F. Locatello, S. Bauer, M. Lucic, G. Raetsch, S. Gelly, B. Schölkopf, and O. Bachem, “Challenging common assumptions in the unsupervised learning of disentangled representations,” vol. 97 of
2019
Later among the works it cites.
M. Ancona, E. Ceolini, C. Öztireli, and M. Gross, “Gradient-based attribution methods,” in
2019
Later among the works it cites.
K. Dhamdhere, M. Sundararajan, and Q. Yan, “How important is a neuron?,” in
2019
Later among the works it cites.
S. Srinivas and F. Fleuret, “Full-gradient representation for neural network visualization,” in
2019
Later among the works it cites.
C.-H. Chang, E. Creager, A. Goldenberg, and D. Duvenaud, “Explaining image classifiers by counterfactual generation,” in
2019
Later among the works it cites.
Accessed: 2020.08.10
M. Turek, “Explainable artificial intelligence (XAI).” · 2020
Closest in time.
Accessed: 2020.08.11
L. Li and Y. Wang, “Manifold: A model-agnostic visual debugging tool for machine learning at Uber.” · 2020
Closest in time.
Accessed: 2020.11.02
J. Larson, S. Mattu, L. Kirchner, and J. Angwin, “How we analyzed the COMPAS recidivism algorithm.” · 2020
Closest in time.
S. Khanna and V. Y. F. Tan, “Economy statistical recurrent units for inferring nonlinear Granger causality,” in
2020
Closest in time.
Altman, D.G., “Analysis of survival times,” in
2020
Closest in time.
T. P. Quinn, D. Nguyen, S. Rana, S. Gupta, and S. Venkatesh, “DeepCoDA: personalized interpretability for compositional health data,” in
2020
Closest in time.
S.-M. Udrescu and M. Tegmark, “AI Feynman: A physics-inspired method for symbolic regression,”
2020
Closest in time.
A. Jha, J. K. Aicher, M. R. Gazzara, D. Singh, and Y. Barash, “Enhanced integrated gradients: improving interpretability of deep learning models using splicing codes as a case study,”
2020
Closest in time.
I. E. Kumar, S. Venkatasubramanian, C. Scheidegger, and S. Friedler, “Problems with Shapley-value-based explanations as feature importance measures,” in
2020
Closest in time.
R. K. Mothilal, A. Sharma, and C. Tan, “Explaining machine learning classifiers through diverse counterfactual explanations,” in
2020
Closest in time.
A.-H. Karimi, G. Barthe, B. Balle, and I. Valera, “Model-agnostic counterfactual explanations for consequential decisions,” in
2020
Closest in time.
A.-H. Karimi, J. von Kügelgen, B. Schölkopf, and I. Valera, “Algorithmic recourse under imperfect causal knowledge: a probabilistic approach,” in
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…