2020

MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering

Longpre, Shayne, Lu, Yi, Daiber, Joachim

Understand

Progress in cross-lingual modeling depends on challenging, realistic, and diverse evaluation sets.

We introduce Multilingual Knowledge Questions and Answers (MKQA), an open-domain question answering evaluation set comprising 10k question-answer pairs aligned across 26 typologically diverse languages (260k question-answer pairs in total).
Answers are based on a heavily curated, language-independent data representation, making results comparable across languages and independent of language-specific passages.
With 26 languages, this dataset supplies the widest range of languages to-date for evaluating question answering.

Built on

Cross-lingual transfer learning for question answering
Original
Chia-Hsuan Lee and Hung-Yi Lee. 2019 · 1907
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c · 1907
Earlier work this paper cites.
Korquad1. 0: Korean qa dataset for machine reading comprehension
Original
Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019 · 1909
Earlier work this paper cites.
BASEBALL: An Automatic Question Answerer . Morgan Kaufmann Publishers Inc., San Francisco, CA, USA
B Green, A Wolf, C Chomsky, and K Laughery. 1986 · 1986
Earlier work this paper cites.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization
Original
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020 · 2003
Earlier work this paper cites.
Unsupervised identification of translationese
Ella Rabinovich and Shuly Wintner. 2015 · 2015
Earlier work this paper cites.
A thorough examination of the CNN/daily mail reading comprehension task
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016 · 2016
Earlier work this paper cites.
Squad: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016
Earlier work this paper cites.
Translationese: Between human and machine translation
Shuly Wintner. 2016 · 2016
Earlier work this paper cites.
Reading Wikipedia to answer open-domain questions
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017 · 2017
Earlier work this paper cites.
Attention-over-attention neural networks for reading comprehension
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017 · 2017
Earlier work this paper cites.
Gated-attention readers for text comprehension
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017 · 2017
Earlier work this paper cites.

Similar

Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017 · 2017
Cited alongside, same era.
Mmqa: A multi-domain multi-lingual question-answering framework for english and hindi
Deepak Gupta, Surabhi Kumari, Asif Ekbal, and Pushpak Bhattacharyya. 2018 · 2018
Cited alongside, same era.
Dureader: a chinese machine reading comprehension dataset from real-world applications
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, et al. 2018 · 2018
Cited alongside, same era.
Drcd: a chinese machine reading comprehension dataset
Original
Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. 2018 · 2018
Cited alongside, same era.
Ethnologue: Languages of the world, twenty
Gary F Simons and Charles D Fennig. 2018 · 2018
Cited alongside, same era.
Cross-lingual machine reading comprehension
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2019a · 2019
Cited alongside, same era.
A span-extraction dataset for chinese machine reading comprehension
Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019b · 2019
Cited alongside, same era.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
Zero-shot reading comprehension by cross-lingual transfer learning with multi-lingual language representation model
Tsung-Yuan Hsu, Chi-Liang Liu, and Hung-yi Lee. 2019 · 2019
Cited alongside, same era.
Bipar: A bilingual parallel dataset for multilingual and cross-lingual reading comprehension on novels
Yimin Jing, Deyi Xiong, and Zhen Yan. 2019 · 2019
Cited alongside, same era.
Cross-lingual training for automatic question generation
Vishwajeet Kumar, Nitish Joshi, Arijit Mukherjee, Ganesh Ramakrishnan, and Preethi Jyothi. 2019 · 2019
Cited alongside, same era.
Natural questions: A benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019 · 2019
Cited alongside, same era.

Then

Neural arabic question answering
Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019 · 2019
Later among the works it cites.
Translation artifacts in cross-lingual transfer learning
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a · 2020
Closest in time.
Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020 · 2020
Closest in time.
Unsupervised cross-lingual representation learning at scale
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020 · 2020
Closest in time.
Dense passage retrieval for open-domain question answering
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020 · 2020
Closest in time.
Mlqa: Evaluating cross-lingual extractive question answering
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020 · 2020
Closest in time.
Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020 · 2020
Closest in time.
How much knowledge can you pack into the parameters of a language model?
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020 · 2020
Closest in time.
Improving massively multilingual neural machine translation and zero-shot translation
Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020 · 2020
Closest in time.
Xor qa: Cross-lingual open-retrieval question answering
Akari Asai, Jungo Kasai, Jonathan H Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021 · 2021
Closest in time.
mt5: A massively multilingual pre-trained text-to-text transformer
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…