2021

To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

Buçinca, Zana, Malaya, Maja Barbara, Gajos, Krzysztof Z.

Understand

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong.

Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it.
Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions.
Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations.

Built on

Dual processes in reasoning?
Peter C Wason and J St BT Evans. 1974 · 1974
Earlier work this paper cites.
A simple sequentially rejective multiple test procedure
Sture Holm. 1979 · 1979
Earlier work this paper cites.
The need for cognition
John T. Cacioppo and Richard E. Petty. 1982 · 1982
Earlier work this paper cites.
Judgment under uncertainty: Heuristics and biases
Daniel Kahneman, Stewart Paul Slovic, Paul Slovic, and Amos Tversky. 1982 · 1982
Earlier work this paper cites.
The efficient assessment of need for cognition
J T Cacioppo, R E Petty, and C F Kao. 1984 · 1984
Earlier work this paper cites.
The Elaboration Likelihood Model of Persuasion
Original
Richard E. Petty and John T. Cacioppo. 1986 · 1986
Earlier work this paper cites.
Need for cognition, causal processing, and memory for behavior
G Daniel Lassiter, Michael A Briggs, and R David Slaw. 1991 · 1991
Earlier work this paper cites.
Need for cognition and advertising: Understanding the role of personality variables in consumer behavior
Curtis P Haugtvedt, Richard E Petty, and John T Cacioppo. 1992 · 1992
Earlier work this paper cites.
Rationality in medical decision making: a review of the literature on doctors’ decision-making biases
Brian H Bornstein and A Christine Emler. 2001 · 2001
Earlier work this paper cites.
An Analysis of the Influence of Need for Cognition on Dynamic Queries Usage. In
Giuseppe Carenini. 2001 · 2001
Earlier work this paper cites.
Understanding differences in web usage: The role of need for cognition and the five factor model of personality
Tracy L Tuten and Michael Bosnjak. 2001 · 2001
Earlier work this paper cites.
Representativeness revisited: Attribute substitution in intuitive judgment
Daniel Kahneman and Shane Frederick. 2002 · 2002
Earlier work this paper cites.
An exploratory cognitive DSS for strategic decision making
Jim Q Chen and Sang M Lee. 2003 · 2003
Earlier work this paper cites.
Cognitive forcing strategies in clinical decisionmaking
Pat Croskerry. 2003a · 2003
Earlier work this paper cites.
The importance of cognitive errors in diagnosis and strategies to minimize them
Pat Croskerry. 2003b · 2003
Earlier work this paper cites.
Matching Health Messages to Information-Processing Styles : Need for Cognition and Mammography Utilization
Pamela Williams-Piehota, Tamera R Schneider, Linda Mowad, and Peter Salovey. 2003 · 2003
Earlier work this paper cites.
Reactance to recommendations: When unsolicited advice yields contrary responses
Gavan J Fitzsimons and Donald R Lehmann. 2004 · 2004
Earlier work this paper cites.
Trust in automation: Designing for appropriate reliance
John D Lee and Katrina A See. 2004 · 2004
Earlier work this paper cites.
Effects of interactivity in a web site: The moderating effect of need for cognition
Maria Sicilia, Salvador Ruiz, and Jose L Munuera. 2005 · 2005
Earlier work this paper cites.
Analysis of unbalanced data by mixed linear models using the MIXED procedure of the SAS system
J Spilke, HP Piepho, and X Hu. 2005 · 2005
Earlier work this paper cites.
Slowing down when you should: a new model of expert judgment
Carol-anne E Moulton, Glenn Regehr, Maria Mylopoulos, and Helen M MacRae. 2007 · 2007
Earlier work this paper cites.
Degradation in spatial knowledge acquisition when using automatic navigation systems. In
Avi Parush, Shir Ahuvia, and Ido Erev. 2007 · 2007
Earlier work this paper cites.
Construction of smoking-relevant risk perceptions among college students: The influence of need for cognition and message content
Jennifer Irvin Vidrine, Vani Nath Simmons, and Thomas H. Brandon. 2007 · 2007
Earlier work this paper cites.

Similar

Overconfidence as a cause of diagnostic error in medicine
Eta S Berner and Mark L Graber. 2008 · 2008
Cited alongside, same era.
When and how diversity benefits teams: The importance of team members’ need for cognition
Eric Kearney, Diether Gebert, and Sven C Voelpel. 2009 · 2009
Cited alongside, same era.
Checklists to reduce diagnostic errors
John W Ely, Mark L Graber, and Pat Croskerry. 2011 · 2011
Cited alongside, same era.
Benefitting infovis with visual difficulties
Jessica Hullman, Eytan Adar, and Priti Shah. 2011 · 2011
Cited alongside, same era.
Thinking, fast and slow
Daniel Kahneman. 2011 · 2011
Cited alongside, same era.
The effects of online reviews on purchasing intention: The moderating role of need for cognition
Chin-Lung Lin, Sheng-Hsien Lee, and Der-Juinn Horng. 2011 · 2011
Cited alongside, same era.
Cognitive interventions to reduce diagnostic error: a narrative review
Mark L Graber, Stephanie Kissam, Velma L Payne, Ashley ND Meyer, Asta Sorensen, Nancy Lenfestey, Elizabeth Tant, Kerm Henriksen, Kenneth LaBresh, and Hardeep Singh. 2012 · 2012
Cited alongside, same era.
Combining human and machine intelligence in large-scale crowdsourcing. In
Ece Kamar, Severin Hacker, and Eric Horvitz. 2012 · 2012
Cited alongside, same era.
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Dale J Barr, Roger Levy, Christoph Scheepers, and Harry J Tily. 2013 · 2013
Cited alongside, same era.
Too much, too little, or just right? Ways explanations impact end users’ mental models. In
Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong. 2013 · 2013
Cited alongside, same era.
Need for cognition and approaches to learning among university students
Ana-Maria Cazan and Simona Elena Indreica. 2014 · 2014
Cited alongside, same era.
Ineffectiveness of cognitive forcing strategies to reduce biases in diagnostic reasoning: a controlled trial
Jonathan Sherbino, Kulamakan Kulasegaram, Elizabeth Howey, and Geoffrey Norman. 2014 · 2014
Cited alongside, same era.

Then

Mental labour
Wouter Kool and Matthew Botvinick. 2018 · 2018
Later among the works it cites.
Good intentions are not enough: how informatics interventions can worsen inequality
Tiffany C Veinot, Hannah Mitchell, and Jessica S Ancker. 2018 · 2018
Later among the works it cites.
Overtrust in the robotic age
Alan R Wagner, Jason Borenstein, and Ayanna Howard. 2018 · 2018
Later among the works it cites.
Beyond accuracy: The role of mental models in human-AI team performance. In
Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019 · 2019
Later among the works it cites.
Cognitive bias research in forensic science: a systematic review
Glinda S Cooper and Vanessa Meterko. 2019 · 2019
Later among the works it cites.
Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom
Louis Deslauriers, Logan S McCarty, Kelly Miller, Kristina Callaghan, and Greg Kestin. 2019 · 2019
Later among the works it cites.
The principles and limits of algorithm-in-the-loop decision making
Ben Green and Yiling Chen. 2019 · 2019
Later among the works it cites.
On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In
Vivian Lai and Chenhao Tan. 2019 · 2019
Later among the works it cites.
To Explain or Not to Explain: The Effects of Personal Characteristics When Explaining Music Recommendations. In
Martijn Millecamp, Nyi Nyi Htun, Cristina Conati, and Katrien Verbert. 2019 · 2019
Later among the works it cites.
A Slow Algorithm Improves Users’ Assessments of the Algorithm’s Accuracy
Joon Sung Park, Rick Barber, Alex Kirlik, and Karrie Karahalios. 2019 · 2019
Later among the works it cites.
The effects of mixing machine learning and human judgment
Michelle Vaccaro and Jim Waldo. 2019 · 2019
Later among the works it cites.
Understanding the effect of accuracy on trust in machine learning models. In
Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. 2019 · 2019
Later among the works it cites.
Evaluating and Aggregating Feature-based Model Explanations. In
Umang Bhatt, Adrian Weller, and José M. F. Moura. 2020 · 2020
Later among the works it cites.
Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. In
Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020 · 2020
Later among the works it cites.
Biased: Uncovering the hidden prejudice that shapes what we see, think, and do
Jennifer L Eberhardt. 2020 · 2020
Later among the works it cites.
"Why is ’Chicago’ Deceptive?" Towards Building Model-Driven Tutorials for Humans. In
Vivian Lai, Han Liu, and Chenhao Tan. 2020 · 2020
Later among the works it cites.
What’s in a User? Towards Personalising Transparency for Music Recommender Interfaces. In
Martijn Millecamp, Nyi Nyi Htun, Cristina Conati, and Katrien Verbert. 2020 · 2020
Later among the works it cites.
Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In
Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020 · 2020
Later among the works it cites.
Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making. In
Yunfeng Zhang, Q. Vera Liao, and Rachel K. E. Bellamy. 2020 · 2020
Later among the works it cites.
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance. In
Gagan Bansal, Tongshuang Wu, Joyce Zhu, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel S Weld. 2021 · 2021
Closest in time.
Explainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021 · 2021
Closest in time.
How machine-learning recommendations influence clinician treatment selections: the example of the antidepressant selection
Maia Jacobs, Melanie F. Pradier, Thomas H. McCoy Jr, Roy H. Perlis, Finale Doshi-Velez, and Krzysztof Z. Gajos. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…