2021

Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge

Bhakthavatsalam, Sumithra, Khashabi, Daniel, Khot, Tushar et al.

Understand

We present the ARC-DA dataset, a direct-answer ("open response", "freeform") version of the ARC (AI2 Reasoning Challenge) multiple-choice dataset.

While ARC has been influential in the community, its multiple-choice format is unrepresentative of real-world questions, and multiple choice formats can be particularly susceptible to artifacts.
The ARC-DA dataset addresses these concerns by converting questions to direct-answer format using a combination of crowdsourcing and expert review.
The resulting dataset contains 2985 questions with a total of 8436 valid answers (questions typically have more than one valid answer).

Built on

Reasoning over paragraph effects in situations
Original
K. Lin, O. Tafjord, P. Clark, and M. Gardner · 1908
Earlier work this paper cites.
A learning algorithm for continually running fully recurrent neural networks
R. J. Williams and D. Zipser · 1989
Earlier work this paper cites.
An information-theoretic approach to automatic evaluation of summaries
C.-Y. Lin, G. Cao, J. Gao, and J.-Y. Nie · 2006
Earlier work this paper cites.
My computer is an honor student – but how intelligent is it? standardized tests as a measure of AI
P. Clark and O. Etzioni · 2016
Earlier work this paper cites.
Combining retrieval, statistics, and inference to answer elementary science questions
P. Clark, O. Etzioni, T. Khot, A. Sabharwal, O. Tafjord, P. D. Turney, and D. Khashabi · 2016
Earlier work this paper cites.
SQuAD: 100,000+ questions for machine comprehension of text
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang · 2016
Earlier work this paper cites.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
M. Joshi, E. Choi, D. S. Weld, and L. S. Zettlemoyer · 2017
Earlier work this paper cites.
A systematic classification of knowledge, reasoning, and context within the ARC dataset
M. Boratko, H. Padigela, D. Mikkilineni, P. Yuvraj, R. Das, A. McCallum, M. Chang, A. Fokoue, P. Kapanipathi, N. Mattei, R. Musa, K. Talamadupula, and M. Witbrock · 2018
Earlier work this paper cites.
Think you have solved question answering? Try ARC, the AI2 Reasoning Challenge
Original
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord · 2018
Earlier work this paper cites.

Similar

Annotation artifacts in natural language inference data
S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith · 2018
Cited alongside, same era.
Can a suit of armor conduct electricity? a new dataset for open book question answering
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal · 2018
Cited alongside, same era.
Modeling naive psychology of characters in simple commonsense stories
H. Rashkin, A. Bosselut, M. Sap, K. Knight, and Y. Choi · 2018
Cited alongside, same era.
HotpotQA: A dataset for diverse, explainable multi-hop question answering
Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning · 2018
Cited alongside, same era.
COMET: Commonsense transformers for automatic knowledge graph construction
A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Celikyilmaz, and Y. Choi · 2019
Cited alongside, same era.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs
D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner · 2019
Cited alongside, same era.
QASC: A dataset for question answering via sentence composition
Original
T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal · 2019
Cited alongside, same era.
Natural Questions: A benchmark for question answering research
T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov · 2019
Cited alongside, same era.
Answering science exam questions using query reformulation with background knowledge
R. Musa, X. Wang, A. Fokoue, N. Mattei, M. Chang, P. Kapanipathi, B. Makni, K. Talamadupula, and M. Witbrock · 2019
Cited alongside, same era.

Then

Learning to attend on essential terms: An enhanced retriever-reader model for open-domain question answering
J. Ni, C. Zhu, W. Chen, and J. McAuley · 2019
Later among the works it cites.
CommonsenseQA: A question answering challenge targeting commonsense knowledge
A. Talmor, J. Herzig, N. Lourie, and J. Berant · 2019
Later among the works it cites.
From recognition to cognition: Visual commonsense reasoning
R. Zellers, Y. Bisk, A. Farhadi, and Y. Choi · 2019
Later among the works it cites.
Unifiedqa: Crossing format boundaries with a single QA system
D. Khashabi, S. Min, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, and H. Hajishirzi · 2020
Later among the works it cites.
Differentiable open-ended commonsense reasoning
Original
B. Y. Lin, H. Sun, B. Dhingra, M. Zaheer, X. Ren, and W. W. Cohen · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu · 2020
Later among the works it cites.
WorldTree V2: A corpus of science-domain structured explanations and inference patterns supporting multi-hop inference
Z. Xie, S. Thiem, J. Martin, E. Wainwright, S. Marmorstein, and P. A. Jansen · 2020
Later among the works it cites.
Temporal reasoning on implicit events from distant supervision
Original
B. Zhou, K. Richardson, Q. Ning, T. Khot, A. Sabharwal, and D. Roth · 2020
Later among the works it cites.
GENIE: A leaderboard for human-in-the-loop evaluation of text generation
Original
D. Khashabi, G. Stanovsky, J. Bragg, N. Lourie, J. Kasai, Y. Choi, N. A. Smith, and D. S. Weld · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…