2021

Investigating the Limitations of Transformers with Simple Arithmetic Tasks

Nogueira, Rodrigo, Jiang, Zhiying, Lin, Jimmy

Understand

The ability to perform arithmetic tasks is a remarkable trait of human intelligence and might form a critical component of more complex reasoning tasks.

In this work, we investigate if the surface form of a number has any influence on how sequence-to-sequence language models learn simple arithmetic tasks such as addition and subtraction across a wide range of values.
We find that how a number is represented in its surface form has a strong influence on the model's accuracy.
In particular, the model fails to learn addition of five-digit numbers when using subwords (e.g., "32"), and it struggles to learn with character-level representations (e.g., "3 2").

Built on

Learning internal representations by error propagation
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1985
Earlier work this paper cites.
Inferring algorithmic patterns with stack-augmented recurrent nets
Armand Joulin and Tomas Mikolov · 2015
Earlier work this paper cites.
Neural GPUs learn algorithms
Original
Łukasz Kaiser and Ilya Sutskever · 2015
Earlier work this paper cites.
Grid long short-term memory
Original
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves · 2015
Earlier work this paper cites.
Extensions and limitations of the neural GPU
Original
Eric Price, Wojciech Zaremba, and Ilya Sutskever · 2016
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Universal transformers
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser · 2018
Earlier work this paper cites.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2018
Earlier work this paper cites.
Deep contextualized word representations
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer · 2018
Earlier work this paper cites.
Analysing mathematical reasoning abilities of neural models
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli · 2018
Earlier work this paper cites.
Neural arithmetic logic units
Andrew Trask, Felix Hill, Scott E. Reed, Jack Rae, Chris Dyer, and Phil Blunsom · 2018
Earlier work this paper cites.
Giving BERT a calculator: Finding operations and arguments with reading comprehension
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler · 2019
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2019
Earlier work this paper cites.
DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner · 2019
Earlier work this paper cites.
Learning numeral embeddings
Original
Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, Haofen Wang, and Kewei Tu · 2019
Earlier work this paper cites.
Deep learning for symbolic mathematics
Guillaume Lample and François Charton · 2019
Earlier work this paper cites.

Similar

Modeling intra-relation in math word problems with different functional multi-head attentions
Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang · 2019
Cited alongside, same era.
Tree-structured decoding for solving math word problems
Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara · 2019
Cited alongside, same era.
Exploring numeracy in word embeddings
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy · 2019
Cited alongside, same era.
NumNet: Machine reading comprehension with numerical reasoning
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu · 2019
Cited alongside, same era.
EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference
Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy · 2019
Cited alongside, same era.
Enhancing the transformer with explicit relational encoding for math problem solving
Original
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao · 2019
Cited alongside, same era.
oLMpics—on what language model pre-training captures
Original
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant · 2019
Cited alongside, same era.
BERT rediscovers the classical NLP pipeline
Ian Tenney, Dipanjan Das, and Ellie Pavlick · 2019
Cited alongside, same era.
Do NLP models know numbers? Probing numeracy in embeddings
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner · 2019
Cited alongside, same era.
Encoding word order in complex embeddings
Benyou Wang, Donghao Zhao, Christina Lioma, Qiuchi Li, Peng Zhang, and Jakob Grue Simonsen · 2019
Cited alongside, same era.
Text2Math: End-to-end parsing text into math expressions
Yanyan Zou and Wei Lu · 2019
Cited alongside, same era.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Cited alongside, same era.

Then

DeBERTa: Decoding-enhanced BERT with disentangled attention
Original
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen · 2020
Later among the works it cites.
Scaling laws for autoregressive generative modeling
Original
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish · 2020
Later among the works it cites.
Improve transformer models with better relative position embeddings
Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang · 2020
Later among the works it cites.
Probing for multilingual numerical understanding in transformer-based language models
Devin Johnson, Denise Mak, Andrew Barker, and Lexi Loessberg-Zahl · 2020
Later among the works it cites.
Rethinking the positional encoding in language pre-training
Original
Guolin Ke, Di He, and Tie-Yan Liu · 2020
Later among the works it cites.
Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models
Original
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren · 2020
Later among the works it cites.
Towards question format independent numerical reasoning: A set of prerequisite tasks
Original
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral · 2020
Later among the works it cites.
The EOS decision and length extrapolation
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D. Manning · 2020
Later among the works it cites.
Generative language modeling for automated theorem proving
Original
Stanislas Polu and Ilya Sutskever · 2020
Later among the works it cites.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu · 2020
Later among the works it cites.
Enhancing the numeracy of word embeddings: A linear algebraic perspective
Yuanhang Ren and Ye Du · 2020
Later among the works it cites.
A sequence-to-sequence approach for numerical slot-filling dialog systems
Hongjie Shi · 2020
Later among the works it cites.
Do language embeddings capture scales?
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth · 2020
Later among the works it cites.
Scaling laws for transfer
Original
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish · 2021
Closest in time.
Representing numbers in NLP: a survey and a vision
Original
Avijit Thawani, Jay Pujara, Pedro A. Szekely, and Filip Ilievski · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…