2020

Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study

Alqaraawi, Ahmed, Schuessler, Martin, Weiß, Philipp et al.

Understand

Convolutional neural networks (CNNs) offer great machine learning performance over a range of applications, but their operation is hard to interpret, even for experts.

Various explanation algorithms have been proposed to address this issue, yet limited research effort has been reported concerning their user evaluation.
In this paper, we report on an online between-group user study designed to evaluate the performance of "saliency maps" - a popular explanation algorithm for image classification applications of CNNs.
Our results indicate that saliency maps produced by the LRP algorithm helped participants to learn about some specific image features the system is sensitive to.

Built on

The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification. In Advances in Neural Information Processing Systems 27 , Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1952–1960
Original
Been Kim, Cynthia Rudin, and Julie A Shah. 2014 · 1960
Earlier work this paper cites.
A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity
Halbert White. 1980 · 1980
Earlier work this paper cites.
The Black Box inside the Glass Box: Presenting Computing Concepts to Novices
Benedict Du Boulay, Tim O’shea, and John Monk. 1999 · 1981
Earlier work this paper cites.
Instructional Manipulation of Users’ Mental Models for Electronic Calculators
Piraye Bayman and Richard E. Mayer. 1984 · 1984
Earlier work this paper cites.
The Role of a Mental Model in Learning to Operate a Device
David E. Kieras and Susan Bovair. 1984 · 1984
Earlier work this paper cites.
Human Issues in the Use of Pattern Recognition Techniques
Alan Dix. 1992 · 1992
Earlier work this paper cites.
A Case for Interaction: A Study of Interactive Information Retrieval Behavior and Effectiveness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’96) . ACM, New York, NY, USA, 205–212
Jürgen Koenemann and Nicholas J. Belkin. 1996 · 1996
Earlier work this paper cites.
Mental Models in Theory and Practice
Neville Moray. 1999 · 1999
Earlier work this paper cites.
Explaining Collaborative Filtering Recommendations. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (CSCW ’00) . ACM, New York, NY, USA, 241–250
Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000 · 2000
Earlier work this paper cites.
Transparent Queries: Investigation Users’ Mental Models of Search Engines. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval . ACM, 217–224
Jack Muramatsu and Wanda Pratt. 2001 · 2001
Earlier work this paper cites.
The Role of Trust in Automation Reliance
Mary T. Dzindolet, Scott A. Peterson, Regina A. Pomranky, Linda G. Pierce, and Hall P. Beck. 2003 · 2003
Earlier work this paper cites.
Trust in Automation: Designing for Appropriate Reliance
John D. Lee and Katrina A. See. 2004 · 2004
Earlier work this paper cites.
Pattern recognition and machine learning
Christopher M Bishop. 2006 · 2006
Earlier work this paper cites.
The PASCAL Visual Object Classes Challenge 2007 (VOC2007)
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2007 · 2007
Earlier work this paper cites.
What do we perceive in a glance of a real-world scene?
Li Fei-Fei, Asha Iyer, Christof Koch, and Pietro Perona. 2007 · 2007
Earlier work this paper cites.
How It Works: A Field Study of Non-Technical Users Interacting with an Intelligent System. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07) . ACM, New York, NY, USA, 31–40
Joe Tullio, Anind K. Dey, Jason Chalecki, and James Fogarty. 2007 · 2007
Earlier work this paper cites.
The Effects of Transparency on Trust in and Acceptance of a Content-Based Art Recommender
Henriette Cramer, Vanessa Evers, Satyan Ramlal, Maarten Someren, Lloyd Rutledge, Natalia Stash, Lora Aroyo, and Bob Wielinga. 2008 · 2008
Earlier work this paper cites.
ImageNet: A Large-Scale Hierarchical Image Database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition . IEEE, Miami, FL, 248–255
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009 · 2009
Earlier work this paper cites.
Support for Context-Aware Intelligibility and Control. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09) . ACM, New York, NY, USA, 859–868
Anind K. Dey and Alan Newberger. 2009 · 2009
Earlier work this paper cites.
Fixing the Program My Computer Learned: Barriers for End Users, Challenges for the Machine. In Proceedings of the 14th International Conference on Intelligent User Interfaces (IUI ’09) . ACM, New York, NY, USA, 187–196
Todd Kulesza, Weng-Keen Wong, Simone Stumpf, Stephen Perona, Rachel White, Margaret M. Burnett, Ian Oberst, and Andrew J. Ko. 2009 · 2009
Earlier work this paper cites.
Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09) . ACM, New York, NY, USA, 2119–2128
Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009 · 2009
Earlier work this paper cites.

Similar

The PASCAL Visual Object Classes Challenge 2012 (VOC2012)
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2012 · 2012
Cited alongside, same era.
Tell Me More?: The Effects of Mental Model Soundness on Personalizing an Intelligent Agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12) . ACM, New York, NY, USA, 1–10
Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012 · 2012
Cited alongside, same era.
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Original
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013 · 2013
Cited alongside, same era.
Learning from a Learning Thermostat: Lessons for Intelligent Systems for the Home. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing . ACM, 93–102
Rayoung Yang and Mark W. Newman. 2013 · 2013
Cited alongside, same era.
Qualitative content analysis: theoretical foundation, basic procedures and software solution
Philipp Mayring. 2014 · 2014
Cited alongside, same era.
On the Relationship between Conceptual and Mental Models. In Gentner et al (e.d) Mental Models
Donald Norman. 2014 · 2014
Cited alongside, same era.
Very Deep Convolutional Networks for Large-Scale Image Recognition
Original
Karen Simonyan and Andrew Zisserman. 2014 · 2014
Cited alongside, same era.
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015 · 2015
Cited alongside, same era.
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. In 2015 International Conference on Healthcare Informatics . 160–169
A. Bussone, S. Stumpf, and D. O’Sullivan. 2015 · 2015
Cited alongside, same era.
Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ’15) . ACM, New York, NY, USA, 126–137
Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015 · 2015
Cited alongside, same era.
Randomized Controlled Field Trials of Predictive Policing
G. O. Mohler, M. B. Short, Sean Malinowski, Mark Johnson, G. E. Tita, Andrea L. Bertozzi, and P. J. Brantingham. 2015 · 2015
Cited alongside, same era.
Tariff Agent: Interacting with a Future Smart Energy System at Home
Alper T. Alan, Enrico Costanza, Sarvapali D. Ramchurn, Joel Fischer, Tom Rodden, and Nicholas R. Jennings. 2016 · 2016
Cited alongside, same era.

Then

Ethically Aligned Design: A Vision for Prioritizing Human Well-Being with Autonomous and Intelligent Systems, Version 2
The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. 2017 · 2017
Later among the works it cites.
Trends and Trajectories for Explainable, Accountable and Intelligible Systems: An HCI Research Agenda. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18) . ACM, Article 582, 18 pages
Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y. Lim, and Mohan Kankanhalli. 2018 · 2018
Later among the works it cites.
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
A. Adadi and M. Berrada. 2018 · 2018
Later among the works it cites.
What Do Different Evaluation Metrics Tell Us About Saliency Models?
Z. Bylinskii, T. Judd, A. Oliva, A. Torralba, and F. Durand. 2019 · 2018
Later among the works it cites.
AI Now 2017 Report
Alex Campolo, Madelyn Sanfilippo, Meredith Whittaker, and Kate Crawford. 2018 · 2018
Later among the works it cites.
A Survey of Methods for Explaining Black Box Models
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018 · 2018
Later among the works it cites.
Human Decisions and Machine Predictions
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018 · 2018
Later among the works it cites.
The Mythos of Model Interpretability
Zachary C. Lipton. 2018 · 2018
Later among the works it cites.
Explanation in Artificial Intelligence: Insights from the Social Sciences
Tim Miller. 2019 · 2018
Later among the works it cites.
How Do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
Original
Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi-Velez. 2018 · 2018
Later among the works it cites.
Manipulating and Measuring Model Interpretability
Original
Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018 · 2018
Later among the works it cites.
A Survey on Deep Learning: Algorithms, Techniques, and Applications
Samira Pouyanfar, Saad Sadiq, Yilin Yan, Haiman Tian, Yudong Tao, Maria Presa Reyes, Mei-Ling Shyu, Shu-Ching Chen, and S. S. Iyengar. 2018 · 2018
Later among the works it cites.
End-to-End Lung Cancer Screening with Three-Dimensional Deep Learning on Low-Dose Chest Computed Tomography
Diego Ardila, Atilla P. Kiraly, Sujeeth Bharadwaj, Bokyung Choi, Joshua J. Reicher, Lily Peng, Daniel Tse, Mozziyar Etemadi, Wenxing Ye, Greg Corrado, David P. Naidich, and Shravya Shetty. 2019 · 2019
Later among the works it cites.
The Effects of Example-Based Explanations in a Machine Learning Interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI ’19) . ACM, New York, NY, USA, 258–262
Carrie J. Cai, Jonas Jongejan, and Jess Holbrook. 2019 · 2019
Later among the works it cites.
What Can AI Do for Me?: Evaluating Machine Learning Interpretations in Cooperative Play. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI ’19) . ACM, New York, NY, USA, 229–239
Shi Feng and Jordan Boyd-Graber. 2019 · 2019
Later among the works it cites.
On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19) . ACM, New York, NY, USA, 29–38
Vivian Lai and Chenhao Tan. 2019 · 2019
Later among the works it cites.
Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. 2019 · 2019
Later among the works it cites.
Minimalistic Explanations: Capturing the Essence of Decisions. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA ’19) . ACM, New York, NY, USA, Article LBW2810, 6 pages
Martin Schuessler and Philipp Weiß. 2019 · 2019
Later among the works it cites.
Progressive Disclosure: Empirically Motivated Approaches to Designing Effective Transparency. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI ’19) . ACM, New York, NY, USA, 107–120
Aaron Springer and Steve Whittaker. 2019 · 2019
Later among the works it cites.
Understanding the Effect of Accuracy on Trust in Machine Learning Models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19) . ACM, New York, NY, USA, 279:1–279:12
Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019 · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…