2021

Representing Numbers in NLP: a Survey and a Vision

Thawani, Avijit, Pujara, Jay, Szekely, Pedro A. et al.

Understand

NLP systems rarely give special consideration to numbers found in text.

This starkly contrasts with the consensus in neuroscience that, in the brain, numbers are represented differently from words.
We arrange recent NLP work on numeracy into a comprehensive taxonomy of tasks and methods.
We break down the subjective notion of numeracy into 7 subtasks, arranged along two dimensions: granularity (exact vs approximate) and units (abstract vs grounded).

Built on

From’f’to’a’on the ny regents science exams: An overview of the aristo project
Original
Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2019 · 1909
Earlier work this paper cites.
Numclaim: Investor’s fine-grained claim detection
Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020 · 1976
Earlier work this paper cites.
The self-organizing map
Teuvo Kohonen. 1990 · 1990
Earlier work this paper cites.
Children’s understanding of counting
Karen Wynn. 1990 · 1990
Earlier work this paper cites.
Wordnet: a lexical database for english
George A Miller. 1995 · 1995
Earlier work this paper cites.
Effective teachers of numeracy
Mike Askew and Mike Askew. 1997 · 1997
Earlier work this paper cites.
Variability signatures distinguish verbal from nonverbal counting for both large and small numbers
Sara Cordes, Rochel Gelman, Charles R Gallistel, and John Whalen. 2001 · 2001
Earlier work this paper cites.
English gigaword
David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2003 · 2003
Earlier work this paper cites.
The biber cognitive estimation test
Sarah E Bullard, Deborah Fein, Mary Kay Gleeson, Nita Tischer, Robert L Mapou, and Edith Kaplan. 2004 · 2004
Earlier work this paper cites.
Bert in negotiations: Early prediction of buyer-seller negotiation outcomes
Original
Kushal Chawla, Gale Lucas, Jonathan Gratch, and Jonathan May. 2020 · 2004
Earlier work this paper cites.
Core systems of number
Lisa Feigenson, Stanislas Dehaene, and Elizabeth Spelke. 2004 · 2004
Earlier work this paper cites.
Core systems in human cognition
Katherine D Kinzler and Elizabeth S Spelke. 2007 · 2007
Earlier work this paper cites.
“was it good? it was provocative.” learning the meaning of scalar adjectives
Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010 · 2010
Earlier work this paper cites.
A fact-aligned corpus of numerical expressions
Sandra Williams and Richard Power. 2010 · 2010
Earlier work this paper cites.
The number sense: How the mind creates mathematics
Stanislas Dehaene. 2011 · 2011
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Original
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 · 2013
Earlier work this paper cites.
GloVe: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014 · 2014
Earlier work this paper cites.
Solving general arithmetic word problems
Subhro Roy and Dan Roth. 2015 · 2015
Earlier work this paper cites.
Reasoning about quantities in natural language
Subhro Roy, Tim Vieira, and Dan Roth. 2015 · 2015
Earlier work this paper cites.
Using the output embedding to improve language models
Original
Ofir Press and Lior Wolf. 2016 · 2016
Earlier work this paper cites.
SQuAD: 100,000+ questions for machine comprehension of text
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016 · 2016
Earlier work this paper cites.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016 · 2016
Earlier work this paper cites.

Similar

Verb physics: Relative physical knowledge of actions and objects
Maxwell Forbes and Yejin Choi. 2017 · 2017
Cited alongside, same era.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Numeral understanding in financial tweets for fine-grained crowd-based forecasting
Chung-Chi Chen, Hen-Hsen Huang, Yow-Ting Shiue, and Hsin-Hsi Chen. 2018 · 2018
Cited alongside, same era.
Deep contextualized word representations
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Cited alongside, same era.
SemEval-2018 task 5: Counting events and participants in the long tail
Marten Postma, Filip Ilievski, and Piek Vossen. 2018 · 2018
Cited alongside, same era.
Grounding gradable adjectives through crowdsourcing
Rebecca Sharp, Mithun Paul, Ajay Nagesh, Dane Bell, and Mihai Surdeanu. 2018 · 2018
Cited alongside, same era.
Numeracy for language models: Evaluating and improving their ability to predict numbers
Original
Georgios P. Spithourakis and Sebastian Riedel. 2018 · 2018
Cited alongside, same era.
Neural arithmetic logic units
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom. 2018 · 2018
Cited alongside, same era.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018 · 2018
Cited alongside, same era.
MathQA: Towards interpretable math word problem solving with operation-based formalisms
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019 · 2019
Cited alongside, same era.
Giving BERT a calculator: Finding operations and arguments with reading comprehension
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019 · 2019
Cited alongside, same era.
Numeracy-600K: Learning numeracy for detecting exaggerated information in market comments
Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen. 2019 · 2019
Cited alongside, same era.

Then

LSTM networks can perform dynamic counting
Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019 · 2019
Later among the works it cites.
Quarel: A dataset and models for answering questions about qualitative relationships
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019 · 2019
Later among the works it cites.
Do NLP models know numbers? probing numeracy in embeddings
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019 · 2019
Later among the works it cites.
Superglue: A stickier benchmark for general-purpose language understanding systems
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019 · 2019
Later among the works it cites.
An empirical investigation of contextualized number prediction
Taylor Berg-Kirkpatrick and Daniel Spokoyny. 2020 · 2020
Later among the works it cites.
On the computational power of transformers and its implications in sequence modeling
Satwik Bhattamishra, Arkil Patel, and Navin Goyal. 2020 · 2020
Later among the works it cites.
Language models are few-shot learners
Original
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020
Later among the works it cites.
Injecting numerical reasoning skills into language models
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020 · 2020
Later among the works it cites.
Learning numeral embedding
Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, and Kewei Tu. 2020 · 2020
Later among the works it cites.
Probing for multilingual numerical understanding in transformer-based language models
Devin Johnson, Denise Mak, Andrew Barker, and Lexi Loessberg-Zahl. 2020 · 2020
Later among the works it cites.
Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020 · 2020
Later among the works it cites.
Contextual datetime language model adaptation for speech recognition
Richard Diehl Martinez, Scott Novotney, Ivan Bulyko, Ariya Rastrow, and Andreas Stolcke. 2020 · 2020
Later among the works it cites.
Towards question format independent numerical reasoning: A set of prerequisite tasks
Original
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral. 2020 · 2020
Later among the works it cites.
Dataset for evaluation of mathematical reasoning abilities in russian
Mikhail Nefedov. 2020 · 2020
Later among the works it cites.
Methods for numeracy-preserving word embeddings
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin. 2020 · 2020
Later among the works it cites.
Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020 · 2020
Later among the works it cites.
Visual sense of number vs. sense of magnitude in humans and machines
Alberto Testolin, Serena Dolfi, Mathijs Rochus, and Marco Zorzi. 2020 · 2020
Later among the works it cites.
Do language embeddings capture scales?
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020 · 2020
Later among the works it cites.
Temporal common sense acquisition with minimal supervision
Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020 · 2020
Later among the works it cites.
Measuring mathematical problem solving with the math dataset
Original
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 · 2021
Closest in time.
Investigating the limitations of the transformers with simple arithmetic tasks
Original
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Li. 2021 · 2021
Closest in time.
Are nlp models really able to solve simple math word problems?
Original
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…