Fetching the paper…

Reading the bibliography…

2020

Neural Arithmetic Units

Madsen, Andreas, Johansen, Alexander Rosenberg

Original
PDF
Post on X

Understand

Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers.

The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication.
We present two new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of a vector.
The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from a vector, when the hidden size is large.

Neural Arithmetic Units · Around

Built on

Differentiable subset sampling
Original
Sang Michael Xie and Stefano Ermon · 1901
Earlier work this paper cites.
Measuring arithmetic extrapolation performance
Original
Andreas Madsen and Alexander R. Johansen · 1910
Earlier work this paper cites.
Probable inference, the law of succession, and statistical inference
Edwin B. Wilson · 1927
Earlier work this paper cites.
Arithmetic in newborn chicks
Rosa Rugani, Laura Fontanari, Eleonora Simoni, Lucia Regolin, and Giorgio Vallortigara · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Neural turing machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba · 2014
Earlier work this paper cites.

Similar

Neural gpus learn algorithms
Original
Lukasz Kaiser and Ilya Sutskever · 2016
Cited alongside, same era.
Grid long short-term memory
Original
Nal Kalchbrenner, Ivo Danihelka, and Alex Graves · 2016
Cited alongside, same era.
The neuronal code for number
Andreas Nieder · 2016
Cited alongside, same era.
Mastering the game of go with deep neural networks and tree search
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis · 2016
Cited alongside, same era.
Improving the neural GPU architecture for algorithm learning
Original
Karlis Freivalds and Renars Liepins · 2017

Then

Neural arithmetic expression calculator
Original
Kaiyu Chen, Yihan Dong, Xipeng Qiu, and Zitian Chen · 2018
Later among the works it cites.
BERT: pre-training of deep bidirectional transformers for language understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Later among the works it cites.
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Brenden M. Lake and Marco Baroni · 2018
Later among the works it cites.
Learning dexterous in-hand manipulation
Original
OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba · 2018
Later among the works it cites.
Neural arithmetic logic units

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

Finding numbers in the brain

Charles R. Gallistel · 2017

Cited alongside, same era.

Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom · 2018

Later among the works it cites.

On evaluating the generalization of LSTM models in formal languages

Mirac Suzgun, Yonatan Belinkov, and Stuart M. Shieber · 2019

Later among the works it cites.