2019

Learning Numeral Embeddings

Jiang, Chengyue, Nian, Zhonglin, Guo, Kaihao et al.

Understand

Word embedding is an essential building block for deep learning methods for natural language processing.

Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored.
Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce.
In this paper, we propose two novel numeral embedding methods that can handle the out-of-vocabulary (OOV) problem for numerals.

Built on

The self-organizing map
Teuvo Kohonen · 1990
Earlier work this paper cites.
Producing high-dimensional semantic spaces from lexical co-occurrence
Kevin Lund and Curt Burgess · 1996
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Placing search in context: The concept revisited
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin · 2001
Earlier work this paper cites.
A neural probabilistic language model
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin · 2003
Earlier work this paper cites.
Three parietal circuits for number processing
Stanislas Dehaene, Manuela Piazza, Philippe Pinel, and Laurent Cohen · 2003
Earlier work this paper cites.
Coding of cognitive magnitude: Compressed scaling of numerical information in the primate prefrontal cortex
Andreas Nieder and Earl K Miller · 2003
Earlier work this paper cites.
Solving logic puzzles: From robust processing to precise semantics
Iddo Lev, Bill MacCartney, Christopher Manning, and Roger Levy · 2004
Earlier work this paper cites.
An improved model of semantic similarity based on lexical co-occurrence
Douglas LT Rohde, Laura M Gonnerman, and David C Plaut · 2006
Earlier work this paper cites.
Extracting semantic representations from word co-occurrence statistics: A computational study
John A Bullinaria and Joseph P Levy · 2007
Earlier work this paper cites.
A unified architecture for natural language processing: Deep neural networks with multitask learning
Ronan Collobert and Jason Weston · 2008
Earlier work this paper cites.
Finding contradictions in text
Marie-Catherine De Marneffe, Anna N Rafferty, and Christopher D Manning · 2008
Earlier work this paper cites.

Similar

Visualizing data using t-sne
Laurens van der Maaten and Geoffrey Hinton · 2008
Cited alongside, same era.
The number sense: How the mind creates mathematics
Stanislas Dehaene · 2011
Cited alongside, same era.
Simple methods for initializing the em algorithm for gaussian mixture models
Johannes Blömer and Kathrin Bujna · 2013
Cited alongside, same era.
Generating sequences with recurrent neural networks
Alex Graves · 2013
Cited alongside, same era.
Word emdeddings through hellinger PCA
Original
Rémi Lebret and Ronan Lebret · 2013
Cited alongside, same era.
Efficient estimation of word representations in vector space
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean · 2013
Cited alongside, same era.
Ncrf++: An open-source neural sequence labeling toolkit
Jie Yang and Yue Zhang · 2013
Cited alongside, same era.
Multimodal distributional semantics
Elia Bruni, Nam Khanh Tran, and Marco Baroni · 2014
Cited alongside, same era.
Simlex-999: Evaluating semantic models with (genuine) similarity estimation
Original
Felix Hill, Roi Reichart, and Anna Korhonen · 2014
Cited alongside, same era.
Glove: Global vectors for word representation
Jeffrey Pennington, Richard Socher, and Christopher Manning · 2014
Cited alongside, same era.
Diamonds in the rough: Event extraction from imperfect microblog data
Ander Intxaurrondo, Eneko Agirre, Oier Lopez De Lacalle, and Mihai Surdeanu · 2015
Cited alongside, same era.
Reasoning about quantities in natural language
Subhro Roy, Tim Vieira, and Dan Roth · 2015
Cited alongside, same era.

Then

Numerical relation extraction with minimal supervision
Aman Madaan, Ashish Mittal, Ganesh Ramakrishnan, Sunita Sarawagi, et al · 2016
Later among the works it cites.
Learning to use formulas to solve simple arithmetic problems
Arindam Mitra and Chitta Baral · 2016
Later among the works it cites.
Solving general arithmetic word problems
Original
Subhro Roy and Dan Roth · 2016
Later among the works it cites.
Numerically grounded language models for semantic error correction
Original
Georgios P. Spithourakis, Isabelle Augenstein, and Sebastian Riedel · 2016
Later among the works it cites.
Learning from explicit and implicit supervision jointly for algebra word problems
Shyam Upadhyay, Ming-Wei Chang, Kai-Wei Chang, and Wen-tau Yih · 2016
Later among the works it cites.
How to evaluate word embeddings? on importance of data efficiency and simple supervised tasks
Original
Stanislaw Jastrzebski, Damian Lesniak, and Wojciech Marian Czarnecki · 2017
Later among the works it cites.
Deep neural solver for math word problems
Yan Wang, Xiaojiang Liu, and Shuming Shi · 2017
Later among the works it cites.
Numeracy for language models: Evaluating and improving their ability to predict numbers
Georgios Spithourakis and Sebastian Riedel · 2018
Later among the works it cites.
Numeracy-600k: Learning numeracy for detecting exaggerated information in market comments
Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen · 2019
Closest in time.
Exploring numeracy in word embeddings
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy · 2019
Closest in time.
Do nlp models know numbers? probing numeracy in embeddings
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…