Understand
The use of language-model-based question-answering systems to aid humans in completing difficult tasks is limited, in part, by the unreliability of the text these systems generate.
- Using hard multiple-choice reading comprehension questions as a testbed, we assess whether presenting humans with arguments for two competing answer options, where one is correct and the other is incorrect, allows human judges to perform more accurately, even when one of the arguments is unreliable and deceptive.
- If this is helpful, we may be able to increase our justified trust in language-model-based systems by asking them to produce these arguments where needed.
- Previous research has shown that just a single turn of arguments in this format is not helpful to humans.
Built on
Quantifying interpretability and trust in machine learning systems
Philipp Schmidt and Felix Biessmann. 2019 · 1901
Earlier work this paper cites.
Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018 · 2018
Earlier work this paper cites.
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Scott M. Lundberg, B. Nair, M. Vavilala, M. Horibe, M. Eisses, Trevor Adams, D. Liston, Daniel King-Wai Low, Shu-Fang Newman, J. Kim, and Su-In Lee. 2018 · 2018
Earlier work this paper cites.
Similar
"Hello AI": Uncovering the onboarding needs of medical practitioners for human-AI collaborative decision-making
Carrie J. Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. 2019 · 2019
Cited alongside, same era.
On human predictions with explanations and predictions of machine learning models: A case study on deception detection
Vivian Lai and Chenhao Tan. 2019 · 2019
Cited alongside, same era.
Does the whole exceed its parts? the effect of ai explanations on complementary team performance
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021 · 2021
Cited alongside, same era.
Then
WebGPT: Browser-assisted question-answering with human feedback
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021 · 2021
Later among the works it cites.
Quality: Question answering with long input texts, yes!
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel R. Bowman. 2022 · 2022
Closest in time.
Single-turn debate does not help humans answer hard reading-comprehension questions
Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, and Samuel Bowman. 2022 · 2022
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…