2018

Neural Arithmetic Logic Units

Trask, Andrew, Hill, Felix, Reed, Scott et al.

Understand

Neural networks can learn to represent and manipulate numerical information, but they seldom generalize well outside of the range of numerical values encountered during training.

To encourage more systematic numerical extrapolation, we propose an architecture that represents numerical quantities as linear activations which are manipulated using primitive arithmetic operators, controlled by learned gates.
We call this module a neural arithmetic logic unit (NALU), by analogy to the arithmetic logic unit in traditional processors.
Experiments show that NALU-enhanced neural networks can learn to track time, perform arithmetic over images of numbers, translate numerical language into real-valued scalars, execute computer code, and count objects in images.

Built on

The child’s understanding of number
Rochel Gelman and C. Randy Gallistel · 1978
Earlier work this paper cites.
Learning representations by back-propagating errors
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams · 1986
Earlier work this paper cites.
Connectionism and cognitive architecture: a critical analysis
Jerry A. Fodor and Zenon W. Pylyshyn · 1988
Earlier work this paper cites.
LSTM recurrent networks learn simple context-free and context-sensitive languages
Felix A Gers and E Schmidhuber · 2001
Earlier work this paper cites.
The Algebraic Mind: Integrating Connectionism and Cognitive Science
Gary F. Marcus · 2003
Earlier work this paper cites.
Tuning curves for approximate numerosity in the human intraparietal sulcus
Manuela Piazza, Véronique Izard, Philippe Pinel, Denis Le Bihan, and Stanislas Dehaene · 2004
Earlier work this paper cites.
Privacy preserving crowd monitoring: Counting people without people models or tracking
Antoni B Chan, Zhang-Sheng John Liang, and Nuno Vasconcelos · 2008
Earlier work this paper cites.
Convolutional deep belief networks on CIFAR-10
Alex Krizhevsky and Geoff Hinton · 2010
Earlier work this paper cites.
The Number Sense: How the Mind Creates Mathematics
Stanislas Dehaene · 2011
Earlier work this paper cites.
Speech recognition with deep recurrent neural networks
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton · 2013
Earlier work this paper cites.
Interactive object counting
Carlos Arteta, Victor Lempitsky, J Alison Noble, and Andrew Zisserman · 2014
Earlier work this paper cites.

Similar

Neural turing machines
Original
Alex Graves, Greg Wayne, and Ivo Danihelka · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Cited alongside, same era.
Learning to execute
Original
Wojciech Zaremba and Ilya Sutskever · 2014
Cited alongside, same era.
VQA: Visual question answering
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh · 2015
Cited alongside, same era.
Learning to transduce with unbounded memory
Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom · 2015
Cited alongside, same era.
Learning to count with deep object features
Original
Santi Seguí, Oriol Pujol, and Jordi Vitrià · 2015
Cited alongside, same era.
Highway networks
Original
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber · 2015
Cited alongside, same era.
Cross-scene crowd counting via deep convolutional neural networks
Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang · 2015
Cited alongside, same era.
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Steven L Brunton, Joshua L Proctor, and J Nathan Kutz · 2016
Cited alongside, same era.
Hybrid computing using a neural network with dynamic external memory
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al · 2016
Cited alongside, same era.
Towards AI-complete question answering: A set of prerequisite toy tasks
Original
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov
Cited in the paper.
Memory networks
Jason Weston, Sumit Chopra, and Antoine Bordes
Cited in the paper.

Then

Identity mappings in deep residual networks
Original
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Later among the works it cites.
Densely connected convolutional networks
Original
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger · 2016
Later among the works it cites.
Exploring the limits of language modeling
Original
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu · 2016
Later among the works it cites.
Ask me anything: Dynamic memory networks for natural language processing
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher · 2016
Later among the works it cites.
Asynchronous methods for deep reinforcement learning
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu · 2016
Later among the works it cites.
Neural programmer-interpreters
Scott E. Reed and Nando de Freitas · 2016
Later among the works it cites.
Finding numbers in the brain
C. Randy Gallistel · 2017
Later among the works it cites.
Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
Brendan Lake and Marco Baroni · 2018
Closest in time.
On the practical computational power of finite precision RNNs for language recognition
Gail Weiss, Yoav Goldberg, and Eran Yahav · 2018
Closest in time.
Microscopy cell counting and detection with fully convolutional regression networks
Weidi Xie, J Alison Noble, and Andrew Zisserman · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…