2018

Explaining Explanations: An Overview of Interpretability of Machine Learning

Gilpin, Leilani H., Bau, David, Yuan, Ben Z. et al.

Understand

There has recently been a surge of work in explanatory artificial intelligence (XAI).

This research area tackles the important problem that complex machines and algorithms often cannot provide insights into their behavior and thought processes.
XAI allows users and parts of the internal system to be more transparent, providing explanations of their decisions in some level of detail.
These explanations are important to ensure algorithmic fairness, identify potential bias/problems in the training data, and to ensure that the algorithms perform as expected.

Built on

P. R. Thagard, “The best explanation: Criteria for theory choice,” The journal of philosophy , vol. 75, no. 2, pp. 76–92, 1978
1978
Earlier work this paper cites.
I. T. Jolliffe, “Principal component analysis and factor analysis,” in Principal component analysis . Springer, 1986, pp. 115–128
1986
Earlier work this paper cites.
H. T. Ng and R. J. Mooney, “The role of coherence in constructing and evaluating abductive explanations,” in Working Notes, AAAI Spring Symposium on Automated Abduction, Stanford, California , 1990
1990
Earlier work this paper cites.
J. F. Reeves, “Computational morality: A process model of belief conflict and resolution for story understanding,” 1991
1991
Earlier work this paper cites.
S. Bromberger, On what we know we don’t know: Explanation, theory, linguistics, and how questions shape them . University of Chicago Press, 1992
1992
Earlier work this paper cites.
G. G. Towell and J. W. Shavlik, “Extracting refined rules from knowledge-based neural networks,” Machine learning , vol. 13, no. 1, pp. 71–101, 1993
1993
Earlier work this paper cites.
D. B. Leake, “Focusing construction and selection of abductive hypotheses,” 1993
1993
Earlier work this paper cites.
S. L. Salzberg, “C4. 5: Programs for machine learning by j. ross quinlan. morgan kaufmann publishers, inc., 1993,” Machine Learning , vol. 16, no. 3, pp. 235–240, 1994
1994
Earlier work this paper cites.
L. Fu, “Rule generation from neural networks,” IEEE Transactions on Systems, Man, and Cybernetics , vol. 24, no. 8, pp. 1114–1124, 1994
1994
Earlier work this paper cites.
R. Andrews, J. Diederich, and A. B. Tickle, “Survey and critique of techniques for extracting rules from trained artificial neural networks,” Knowledge-based systems , vol. 8, no. 6, pp. 373–389, 1995
1995
Earlier work this paper cites.
S. Thrun, “Extracting rules from artificial neural networks with distributed representations,” in Advances in neural information processing systems , 1995, pp. 505–512
1995
Earlier work this paper cites.
M. W. Craven, “Extracting comprehensible models from trained neural networks,” Ph.D. dissertation, University of Wisconsin, Madison, 1996
1996
Earlier work this paper cites.
J. M. Benítez, J. L. Castro, and I. Requena, “Are artificial neural networks black boxes?” IEEE Transactions on neural networks , vol. 8, no. 5, pp. 1156–1164, 1997
1997
Earlier work this paper cites.
G. P. Schmitz, C. Aldrich, and F. S. Gouws, “Ann-dt: an algorithm for extraction of decision trees from artificial neural networks,” IEEE Transactions on Neural Networks , vol. 10, no. 6, pp. 1392–1401, 1999
1999
Earlier work this paper cites.
I. A. Taha and J. Ghosh, “Symbolic interpretation of artificial neural networks,” IEEE Transactions on knowledge and data engineering , vol. 11, no. 3, pp. 448–463, 1999
1999
Earlier work this paper cites.
H. Tsukimoto, “Extracting rules from trained neural networks,” IEEE Transactions on Neural Networks , vol. 11, no. 2, pp. 377–389, 2000
2000
Earlier work this paper cites.
R. Setiono and W. K. Leow, “Fernn: An algorithm for fast extraction of rules from neural networks,” Applied Intelligence , vol. 12, no. 1-2, pp. 15–25, 2000
2000
Earlier work this paper cites.
A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural networks , vol. 13, no. 4-5, pp. 411–430, 2000
2000
Earlier work this paper cites.
M. Sato and H. Tsukimoto, “Rule extraction from neural networks via decision tree induction,” in Neural Networks, 2001. Proceedings. IJCNN’01. International Joint Conference on , vol. 3. IEEE, 2001, pp. 1870–1875
2001
Earlier work this paper cites.
U. Johansson, R. Konig, and L. Niklasson, “Automatically balancing accuracy and comprehensibility in predictive modeling,” in Information Fusion, 2005 8th International Conference on , vol. 2. IEEE, 2005, pp. 7–pp
2005
Earlier work this paper cites.
M. W. Berry, M. Browne, A. N. Langville, V. P. Pauca, and R. J. Plemmons, “Algorithms and applications for approximate nonnegative matrix factorization,” Computational statistics & data analysis , vol. 52, no. 1, pp. 155–173, 2007
2007
Earlier work this paper cites.
D. Boyd and K. Crawford, “Critical questions for big data: Provocations for a cultural, technological, and scholarly phenomenon,” Information, communication & society , vol. 15, no. 5, pp. 662–679, 2012
2012
Earlier work this paper cites.
M. G. Augasta and T. Kathirvalavakumar, “Reverse engineering the neural networks for rule extraction in classification problems,” Neural processing letters , vol. 35, no. 2, pp. 131–150, 2012
2012
Earlier work this paper cites.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199 , 2013
Original
2013
Earlier work this paper cites.
K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034 , 2013
Original
2013
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
J. Koster-Hale and R. Saxe, “Theory of mind: a neural prediction problem,” Neuron , vol. 79, no. 5, pp. 836–848, 2013
2013
Earlier work this paper cites.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
Original
2014
Earlier work this paper cites.

Similar

M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European conference on computer vision . Springer, 2014, pp. 818–833
2014
Cited alongside, same era.
A. S. Razavian, H. Azizpour, J. Sullivan, and S. Carlsson, “Cnn features off-the-shelf: an astounding baseline for recognition,” in Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on . IEEE, 2014, pp. 512–519
2014
Cited alongside, same era.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” in Advances in neural information processing systems , 2014, pp. 3320–3328
2014
Cited alongside, same era.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object detectors emerge in deep scene cnns,” arXiv preprint arXiv:1412.6856 , 2014
Original
2014
Cited alongside, same era.
A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2015, pp. 427–436
2015
Cited alongside, same era.
S. Russell, D. Dewey, and M. Tegmark, “Research priorities for robust and beneficial artificial intelligence,” Ai Magazine , vol. 36, no. 4, pp. 105–114, 2015
2015
Cited alongside, same era.
S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation,” PloS one , vol. 10, no. 7, p. e0130140, 2015
2015
Cited alongside, same era.
T. Xiao, Y. Xu, K. Yang, J. Zhang, Y. Peng, and Z. Zhang, “The application of two-level attention models in deep convolutional neural network for fine-grained image classification,” in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on . IEEE, 2015, pp. 842–850
2015
Cited alongside, same era.
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision , 2015, pp. 2425–2433
2015
Cited alongside, same era.
R. W. Magid, M. Sheskin, and L. E. Schulz, “Imagination and the generation of new ideas,” Cognitive Development , vol. 34, pp. 99–110, 2015
2015
Cited alongside, same era.
J. Angwin, J. Larson, S. Mattu, and L. Kirchner, “Machine bias,” https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [Accessed May 24, 2019], 2016
2016
Cited alongside, same era.
N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in IEEE European Symposium on Security and Privacy (EuroS&P) , 2016, pp. 372–387
2016
Cited alongside, same era.

Then

Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” 2017
2017
Later among the works it cites.
N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security , 2017, pp. 3–14
2017
Later among the works it cites.
A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science , vol. 356, no. 6334, pp. 183–186, 2017
2017
Later among the works it cites.
B. Herman, “The promise and peril of human evaluation for model interpretability,” arXiv preprint arXiv:1711.07414 , 2017
Original
2017
Later among the works it cites.
A. Shrikumar, P. Greenside, and A. Kundaje, “Learning important features through propagating activation differences,” arXiv preprint arXiv:1704.02685 , 2017
Original
2017
Later among the works it cites.
M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” arXiv preprint arXiv:1703.01365 , 2017
Original
2017
Later among the works it cites.
D. Smilkov, N. Thorat, B. Kim, F. B. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” CoRR , vol. abs/1706.03825, 2017. [Online]. Available: http://arxiv.org/abs/1706.03825
Original
2017
Later among the works it cites.
M. Ancona, E. Ceolini, A. C. Öztireli, and M. H. Gross, “A unified view of gradient-based attribution methods for deep neural networks,” CoRR , vol. abs/1711.06104, 2017. [Online]. Available: http://arxiv.org/abs/1711.06104
Original
2017
Later among the works it cites.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba, “Network dissection: Quantifying interpretability of deep visual representations,” in Computer Vision and Pattern Recognition , 2017
2017
Later among the works it cites.
B. Kim, J. Gilmer, F. Viegas, U. Erlingsson, and M. Wattenberg, “Tcav: Relative concept importance testing with linear concept activation vectors,” arXiv preprint arXiv:1711.11279 , 2017
Original
2017
Later among the works it cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017, pp. 6000–6010
2017
Later among the works it cites.
A. Das, H. Agrawal, L. Zitnick, D. Parikh, and D. Batra, “Human attention in visual question answering: Do humans and deep networks look at the same regions?” Computer Vision and Image Understanding , vol. 163, pp. 90–100, 2017
2017
Later among the works it cites.
A. S. Ross, M. C. Hughes, and F. Doshi-Velez, “Right for the right reasons: Training differentiable models by constraining their explanations,” arXiv preprint arXiv:1703.03717 , 2017
Original
2017
Later among the works it cites.
Q. Zhang, R. Cao, Y. N. Wu, and S.-C. Zhu, “Growing interpretable part graphs on convnets via multi-shot learning.” 2017
2017
Later among the works it cites.
S. Sabour, N. Frosst, and G. E. Hinton, “Dynamic routing between capsules,” in Advances in Neural Information Processing Systems , 2017, pp. 3859–3869
2017
Later among the works it cites.
F. Doshi-Velez and B. Kim, “Towards a rigorous science of interpretable machine learning,” arXiv , 2017. [Online]. Available: https://arxiv.org/abs/1702.08608
Original
2017
Later among the works it cites.
M. Fox, D. Long, and D. Magazzeni, “Explainable planning,” CoRR , vol. abs/1709.10256, 2017. [Online]. Available: http://arxiv.org/abs/1709.10256
Original
2017
Later among the works it cites.
P. Winston and D. Holmes, “The genesis manifesto: Story understanding and human intelligence,” 2017
2017
Later among the works it cites.
Y. LeCun, “My take on ali rahimi’s test of time award talk at nips,” 2017
2017
Later among the works it cites.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” 2018
2018
Closest in time.
A. D. Kraft, “Vision by alignment,” Ph.D. dissertation, MIT, 77 Massachusetts Ave., 2 2018, an optional note
2018
Closest in time.
Q.-s. Zhang and S.-C. Zhu, “Visual interpretability for deep learning: a survey,” Frontiers of Information Technology & Electronic Engineering , vol. 19, no. 1, pp. 27–39, 2018
2018
Closest in time.
J. Frankle and M. Carbin, “The lottery ticket hypothesis: Training pruned neural networks,” CoRR , vol. abs/1803.03635, 2018. [Online]. Available: http://arxiv.org/abs/1803.03635
Original
2018
Closest in time.
D. H. Park, L. A. Hendricks, Z. Akata, A. Rohrbach, B. Schiele, T. Darrell, and M. Rohrbach, “Multimodal explanations: Justifying decisions and pointing to the evidence,” CoRR , vol. abs/1802.08129, 2018. [Online]. Available: http://arxiv.org/abs/1802.08129
Original
2018
Closest in time.
Q. Zhang, Y. N. Wu, and S.-C. Zhu, “Interpretable convolutional neural networks,” 2018
2018
Closest in time.
Q. Zhang, Y. Yang, Y. Liu, Y. N. Wu, and S.-C. Zhu, “Unsupervised learning of neural networks to explain neural networks,” arXiv preprint arXiv:1805.07468 , 2018
Original
2018
Closest in time.
A. Abdul, J. Vermeulen, D. Wang, B. Y. Lim, and M. Kankanhalli, “Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda,” in Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems . ACM, 2018, p. 582
2018
Closest in time.
R. Guidotti, A. Monreale, F. Turini, D. Pedreschi, and F. Giannotti, “A survey of methods for explaining black box models,” arXiv preprint arXiv:1802.01933 , 2018
Original
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…