2018

Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge

Mihaylov, Todor, Frank, Anette

Understand

We introduce a neural reading comprehension model that integrates external commonsense knowledge, encoded as a key-value memory, in a cloze-style setting.

Instead of relying only on document-to-question interaction or discrete features as in prior work, our model attends to relevant external knowledge and combines this knowledge with the context representation before inferring the answer.
This allows the model to attract and imply knowledge from an external knowledge source that is not explicitly stated in the text, but that is relevant for inferring the answer.
Our model improves results over a very strong baseline on a hard Common Nouns dataset, making it a strong competitor of much more complex models.

Built on

Dropout: A simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014 · 1958
Earlier work this paper cites.
Introduction to wordnet: An on-line lexical database*
George A. Miller, Richard Beckwith, Christiane Fellbaum, Derek Gross, and Katherine J. Miller. 1990 · 1990
Earlier work this paper cites.
Open mind common sense: Knowledge acquisition from the general public
Parmjit Singh, T Lin, E.T. Mueller, G Lim, T Perkins, and W.L. Zhu. 2002 · 2002
Earlier work this paper cites.
Open mind common sense: Knowledge acquisition from the general public
Parmjit Singh, T Lin, E.T. Mueller, G Lim, T Perkins, and W.L. Zhu. 2002 · 2002
Earlier work this paper cites.
Identifying relations for open information extraction
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011 · 2011
Earlier work this paper cites.
Open language learning for information extraction
Mausam, Michael Schmitz, Stephen Soderland, Robert Bart, and Oren Etzioni. 2012 · 2012
Earlier work this paper cites.
BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network
Roberto Navigli and Simone Paolo Ponzetto. 2012 · 2012
Earlier work this paper cites.
Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia
Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012 · 2012
Earlier work this paper cites.
Translating embeddings for modeling multi-relational data
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013 · 2013
Earlier work this paper cites.
MCTest: A challenge dataset for the open-domain machine comprehension of text
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013 · 2013
Earlier work this paper cites.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
Junyoung Chung, Çağlar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014 · 2014
Earlier work this paper cites.
Teaching machines to read and comprehend
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015 · 2015
Earlier work this paper cites.
The goldilocks principle: Reading children’s books with explicit memory representations
Original
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015 · 2015
Earlier work this paper cites.
Never-ending learning
T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Betteridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel, J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed, N. Nakashole, E. Platanios, A. Ritter, M. Samadi, B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen, A. Saparov, M. Greaves, and J. Welling. 2015 · 2015
Earlier work this paper cites.
Memory networks
Jason Weston, Sumit Chopra, and Antoine Bordes. 2015 · 2015
Earlier work this paper cites.
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
Original
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014 · 2015
Earlier work this paper cites.
TensorFlow: Large-scale machine learning on heterogeneous systems
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015 · 2015
Earlier work this paper cites.

Similar

The goldilocks principle: Reading children’s books with explicit memory representations
Original
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015 · 2015
Cited alongside, same era.
Adam: a Method for Stochastic Optimization
Diederik P. Kingma and Jimmy Lei Ba. 2015 · 2015
Cited alongside, same era.
A neural knowledge language model
Original
Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. 2016 · 2016
Cited alongside, same era.
Nested propositions in open information extraction
Nikita Bhutani, H V Jagadish, and Dragomir Radev. 2016 · 2016
Cited alongside, same era.
A thorough examination of the cnn/daily mail reading comprehension task
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016 · 2016
Cited alongside, same era.
Text understanding with the attention sum reader network
Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016 · 2016
Cited alongside, same era.
Globally coherent text generation with neural checklist models
Chloé Kiddon, Luke Zettlemoyer, and Yejin Choi. 2016 · 2016
Cited alongside, same era.
Key-value memory networks for directly reading documents
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016 · 2016
Cited alongside, same era.
Who did what: A large-scale person-centered cloze dataset
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016 · 2016
Cited alongside, same era.
Squad: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016
Cited alongside, same era.
Reasonet: Learning to stop reading in machine comprehension
Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. 2016 · 2016
Cited alongside, same era.
Iterative alternating neural attention for machine reading
Original
Alessandro Sordoni, Phillip Bachman, and Yoshua Bengio. 2016 · 2016
Cited alongside, same era.

Then

Natural language comprehension with the epireader
Adam Trischler, Zheng Ye, Xingdi Yuan, Philip Bachman, Alessandro Sordoni, and Kaheer Suleman. 2016 · 2016
Later among the works it cites.
Attention-over-attention neural networks for reading comprehension
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017 · 2017
Later among the works it cites.
Gated-attention readers for text comprehension
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017 · 2017
Later among the works it cites.
Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning
Shizhu He, Cao Liu, Kang Liu, and Jun Zhao. 2017 · 2017
Later among the works it cites.
Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017 · 2017
Later among the works it cites.
World knowledge for reading comprehension: Rare entity prediction with hierarchical lstms using external descriptions
Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. 2017 · 2017
Later among the works it cites.
Reasoning with Memory Augmented Neural Networks for Language Comprehension
Original
Tsendsuren Munkhdalai and Hong Yu. 2016 · 2017
Later among the works it cites.
Bi-Directional Attention Flow for Machine Comprehension
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hananneh Hajishirzi. 2017 · 2017
Later among the works it cites.
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Robert Speer, Joshua Chin, and Catherine Havasi. 2017 · 2017
Later among the works it cites.
Newsqa: A machine comprehension dataset
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017 · 2017
Later among the works it cites.
Gated self-matching networks for reading comprehension and question answering
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017 · 2017
Later among the works it cites.
Dynamic integration of background knowledge in neural NLU systems
Original
Dirk Weissenborn, Tomas Kocisky, and Chris Dyer. 2017 · 2017
Later among the works it cites.
Dynamic coattention networks for question answering
Original
Caiming Xiong, Victor Zhong, and Richard Socher. 2016 · 2017
Later among the works it cites.
Leveraging knowledge bases in lstms for improving machine reading
Bishan Yang and Tom Mitchell. 2017 · 2017
Later among the works it cites.
Reference-aware language models
Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2017 · 2017
Later among the works it cites.
Attention-over-attention neural networks for reading comprehension
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017 · 2017
Later among the works it cites.
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Robert Speer, Joshua Chin, and Catherine Havasi. 2017 · 2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…