2021

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Min, Sewon, Boyd-Graber, Jordan, Alberti, Chris et al.

Understand

We review the EfficientQA competition from NeurIPS 2020.

The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers.
The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets.
These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models.

Built on

Quizbowl: The case for incremental question answering
Original
Pedro Rodriguez, Shi Feng, Mohit Iyyer, He He, and Jordan Boyd-Graber · 1904
Earlier work this paper cites.
What question answering can learn from trivia nerds
Original
Jordan Boyd-Graber and Benjamin Börschinger · 1910
Earlier work this paper cites.
Computers & thought
Alan M. Turing · 1995
Earlier work this paper cites.
Building a question answering test collection
Ellen M Voorhees and Dawn M Tice · 2000
Earlier work this paper cites.
Brainiac: adventures in the curious, competitive, compulsive world of trivia buffs
Ken Jennings · 2006
Earlier work this paper cites.
Building Watson: An Overview of the DeepQA Project
David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty · 2010
Earlier work this paper cites.
Besting the quiz master: Crowdsourcing incremental classification games
Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daume III · 2012
Earlier work this paper cites.
Opponent modeling in deep reinforcement learning
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III · 2016
Earlier work this paper cites.
SQuAD: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang · 2016
Earlier work this paper cites.
Reading Wikipedia to answer open-domain questions
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes · 2017
Earlier work this paper cites.
Adversarial examples for evaluating reading comprehension systems
Robin Jia and Percy Liang · 2017
Earlier work this paper cites.
Billion-scale similarity search with GPUs
Original
Jeff Johnson, Matthijs Douze, and Hervé Jégou · 2017
Earlier work this paper cites.
TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer · 2017
Earlier work this paper cites.
Robots can now read better than humans, putting millions of jobs at risk, 2018
Anthony Cuthbertson · 2018
Earlier work this paper cites.
How much reading does reading comprehension require? a critical investigation of popular benchmarks
Divyansh Kaushik and Zachary C. Lipton · 2018
Earlier work this paper cites.
Know what you don’t know: Unanswerable questions for SQuAD
Pranav Rajpurkar, Robin Jia, and Percy Liang · 2018
Earlier work this paper cites.
What makes reading comprehension questions easier?
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa · 2018
Earlier work this paper cites.

Similar

Synthetic QA corpora generation with roundtrip consistency
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins · 2019
Cited alongside, same era.
Learning to retrieve reasoning paths over wikipedia graph for question answering
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong · 2019
Cited alongside, same era.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner · 2019
Cited alongside, same era.
Eli5: Long form question answering
Original
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli · 2019
Cited alongside, same era.
What AI can do for me: Evaluating machine learning interpretations in cooperative play
Shi Feng and Jordan Boyd-Graber · 2019
Cited alongside, same era.
Technical report on conversational question answering, 2019
Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu · 2019
Cited alongside, same era.
Natural Questions: a benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov · 2019
Cited alongside, same era.
Latent retrieval for weakly supervised open domain question answering
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova · 2019
Cited alongside, same era.
Unsupervised question answering by cloze translation
Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel · 2019
Cited alongside, same era.
RoBERTa: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov · 2019
Cited alongside, same era.
Real-time open-domain question answering with dense-sparse phrase index
Minjoon Seo, Jinhyuk Lee, Tom Kwiatkowski, Ankur Parikh, Ali Farhadi, and Hannaneh Hajishirzi · 2019
Cited alongside, same era.
Trick me if you can: Human-in-the-loop generation of adversarial question answering examples
Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber · 2019
Cited alongside, same era.

Then

Rethinking the objectives of extractive question answering
Original
Martin Fajcik, Josef Jon, Santosh Kesiraju, and Pavel Smrz · 2020
Later among the works it cites.
REALM: Retrieval-augmented language model pre-training
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang · 2020
Later among the works it cites.
Leveraging passage retrieval with generative models for open domain question answering
Original
Gautier Izacard and Edouard Grave · 2020
Later among the works it cites.
A memory efficient baseline for open domain question answering, 2020
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave · 2020
Later among the works it cites.
Dense passage retrieval for open-domain question answering
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih · 2020
Later among the works it cites.
ALBERT: A lite BERT for self-supervised learning of language representations
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut · 2020
Later among the works it cites.
Generation-augmented retrieval for open-domain question answering
Original
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen · 2020
Later among the works it cites.
AmbigQA: Answering ambiguous open-domain questions
Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer · 2020
Later among the works it cites.
Unified open-domain question answering with structured and unstructured knowledge
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen-tau Yih · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu · 2020
Later among the works it cites.
How much knowledge can you pack into the parameters of a language model?
Adam Roberts, Colin Raffel, and Noam Shazeer · 2020
Later among the works it cites.
MobileBERT: a compact task-agnostic bert for resource-limited devices
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou · 2020
Later among the works it cites.
Is retriever merely an approximator of reader?
Original
Sohee Yang and Minjoon Seo · 2020
Later among the works it cites.
Unitedqa: A hybrid approach for open domain question answering
Original
Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao · 2021
Closest in time.
Pruning the index contents for memory efficient open-domain qa
Original
Martin Fajcik, Martin Docekal, Karel Ondrej, and Pavel Smrz · 2021
Closest in time.
Paq: 65 million probably-asked questions and what you can do with them
Original
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel · 2021
Closest in time.
Designing a minimal retrieve-and-read system for open-domain question answering
Sohee Yang and Minjoon Seo · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…