2020

Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models

Lin, Bill Yuchen, Lee, Seyeon, Khanna, Rahul et al.

Understand

Recent works show that pre-trained language models (PTLMs), such as BERT, possess certain commonsense and factual knowledge.

They suggest that it is promising to use PTLMs as "neural knowledge bases" via predicting masked words.
Surprisingly, we find that this may not work for numerical commonsense knowledge (e.g., a bird usually has two legs).
In this paper, we investigate whether and to what extent we can induce numerical commonsense knowledge from PTLMs as well as the robustness of this process.

Built on

RoBERTa: A robustly optimized bert pretraining approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
CommonGen: A constrained text generation dataset towards generative commonsense reasoning
Original
Bill Yuchen Lin, Ming Shen, Yu Xing, Pei Zhou, and Xiang Ren. 2019b · 1911
Earlier work this paper cites.
olmpics - on what language model pre-training captures
Original
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019a · 1912
Earlier work this paper cites.
Open mind common sense: Knowledge acquisition from the general public
Push Singh, Thomas Lin, Erik T Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. 2002 · 2002
Earlier work this paper cites.
Injecting numerical reasoning skills into language models
Original
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020 · 2004
Earlier work this paper cites.
Genericskb: A knowledge base of generic statements
Original
Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter Clark. 2020 · 2005
Earlier work this paper cites.
Microsoft coco: Common objects in context
Original
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014 · 2014
Earlier work this paper cites.
MAWPS: A math word problem repository
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016 · 2016
Earlier work this paper cites.
Verb physics: Relative physical knowledge of actions and objects
Maxwell Forbes and Yejin Choi. 2017 · 2017
Earlier work this paper cites.
Conceptnet 5.5: An open multilingual graph of general knowledge
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017 · 2017
Earlier work this paper cites.

Similar

What does BERT look at? an analysis of BERT’s attention
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019 · 2019
Cited alongside, same era.
Commonsense knowledge mining from pretrained models
Joe Davison, Joshua Feldman, and Alexander Rush. 2019 · 2019
Cited alongside, same era.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019 · 2019
Cited alongside, same era.
How large are lions? inducing distributions over quantitative attributes
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019 · 2019
Cited alongside, same era.
How pre-trained word representations capture commonsense physical comparisons
Pranav Goel, Shi Feng, and Jordan Boyd-Graber. 2019 · 2019
Cited alongside, same era.
Natural questions: A benchmark for question answering research
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019 · 2019
Cited alongside, same era.
KagNet: Knowledge-aware graph networks for commonsense reasoning
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019a · 2019
Cited alongside, same era.
Language models as knowledge bases?
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019 · 2019
Cited alongside, same era.
Do language models have common sense
Trieu H. Trinh and Quoc V. Le. 2018a
Cited in the paper.
A simple method for commonsense reasoning
Original
Trieu H. Trinh and Quoc V. Le. 2018b
Cited in the paper.

Then

CommonsenseQA: A question answering challenge targeting commonsense knowledge
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019b · 2019
Later among the works it cites.
BERT rediscovers the classical NLP pipeline
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019 · 2019
Later among the works it cites.
Do NLP models know numbers? probing numeracy in embeddings
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019 · 2019
Later among the works it cites.
Vatex: A large-scale, high-quality multilingual dataset for video-and-language research
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019 · 2019
Later among the works it cites.
Text2Math: End-to-end parsing text into math expressions
Yanyan Zou and Wei Lu. 2019 · 2019
Later among the works it cites.
Learning to retrieve reasoning paths over wikipedia graph for question answering
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020 · 2020
Closest in time.
Inducing relational knowledge from bert
Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020 · 2020
Closest in time.
Measuring numerical common sense: Is a word embedding approach effective?
Hiroaki Yamane, Chin-Yew Lin, and Tatsuya Harada. 2020 · 2020
Closest in time.
Evaluating commonsense in pre-trained language models
Xuhui Zhou, Y. Zhang, Leyang Cui, and Dandan Huang. 2020 · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…