2020

Learning advanced mathematical computations from examples

Charton, François, Hayat, Amaury, Lample, Guillaume

Understand

Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such as local stability, behavior at infinity and controllability.

We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system.
This demonstrates that neural networks can learn to perform complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.

Built on

Automatic steering tests
Nicolas Minorsky · 1930
Earlier work this paper cites.
The unreasonable effectiveness of mathematics in the natural sciences
Eugene P Wigner · 1960
Earlier work this paper cites.
Controllability of linear dynamical systems
Rudolf E. Kalman, Yu-Chi Ho, and Kumpati S. Narendra · 1963
Earlier work this paper cites.
Stabilizability and optimal control
Dahlard L Lukes · 1968
Earlier work this paper cites.
An easy way to stabilize a linear constant system
David Kleinman · 1970
Earlier work this paper cites.
Approximation by superpositions of a sigmoidal function
George Cybenko · 1989
Earlier work this paper cites.
Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks
Kurt Hornik, Maxwell Stinchcombe, and Halbert White · 1990
Earlier work this paper cites.
Neural algorithm for solving differential equations
Hyuk Lee and In Seok Kang · 1990
Earlier work this paper cites.
Approximation capabilities of multilayer feedforward networks
Kurt Hornik · 1991
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Artificial neural networks for solving ordinary and partial differential equations
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis · 1998
Earlier work this paper cites.
Approximation theory of the mlp model in neural networks
Allan Pinkus · 1999
Earlier work this paper cites.
Neural-network methods for boundary value problems with irregular boundaries
Isaac E Lagaris, Aristidis C Likas, and Dimitris G Papageorgiou · 2000
Earlier work this paper cites.
Control and nonlinearity , volume 136 of Mathematical Surveys and Monographs
Jean-Michel Coron · 2007
Earlier work this paper cites.

Similar

The Tacit Dimension
Michael Polanyi and Amartya Sen · 2009
Cited alongside, same era.
Partial differential equations , volume 19
Lawrence C Evans · 2010
Cited alongside, same era.
Fourier analysis and nonlinear partial differential equations , volume 343
Hajer Bahouri, Jean-Yves Chemin, and Raphaël Danchin · 2011
Cited alongside, same era.
Solving partial differential equations using artificial neural networks
Keith Rudd · 2013
Cited alongside, same era.
Neural machine translation by jointly learning to align and translate
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio · 2014
Cited alongside, same era.
Adam: A method for stochastic optimization
Original
Diederik P Kingma and Jimmy Ba · 2014
Cited alongside, same era.
Sequence to sequence learning with neural networks
Ilya Sutskever, Oriol Vinyals, and Quoc V Le · 2014
Cited alongside, same era.
Learning to execute
Original
Wojciech Zaremba and Ilya Sutskever · 2014
Cited alongside, same era.
Neural gpus learn algorithms
Original
Lukasz Kaiser and Ilya Sutskever · 2015
Cited alongside, same era.
Improved semantic representations from tree-structured long short-term memory networks
Original
Kai Sheng Tai, Richard Socher, and Christopher D Manning · 2015
Cited alongside, same era.
Collision avoidance and stabilization for autonomous vehicles in emergency scenarios
Joseph Funke, Matthew Brown, Stephen M Erlien, and J Christian Gerdes · 2016
Cited alongside, same era.
Bag of tricks for efficient text classification
Original
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov · 2016
Cited alongside, same era.

Then

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin · 2017
Later among the works it cites.
Can neural networks understand logical entailment?
Original
Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette · 2018
Later among the works it cites.
Optimal approximation of piecewise smooth functions using deep relu neural networks
Philipp Petersen and Felix Voigtlaender · 2018
Later among the works it cites.
Learning a sat solver from single-bit supervision
Original
Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L Dill · 2018
Later among the works it cites.
Dgm: A deep learning algorithm for solving partial differential equations
Justin Sirignano and Konstantinos Spiliopoulos · 2018
Later among the works it cites.
Neural arithmetic logic units
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom · 2018
Later among the works it cites.
What does bert look at? an analysis of bert’s attention
Original
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning · 2019
Later among the works it cites.
Matlab optimization toolbox (r2019a), 2019
MathWorks · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Later among the works it cites.
Analysing mathematical reasoning abilities of neural models
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli · 2019
Later among the works it cites.
Mathematica, version 12.0, 2019
Wolfram-Research · 2019
Later among the works it cites.
Deep learning for symbolic mathematics
Guillaume Lample and François Charton · 2020
Closest in time.
SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1. 0 Contributors · 2020
Closest in time.
It’s not what machines can learn, it’s what we cannot teach
Original
Gal Yehuda, Moshe Gabel, and Assaf Schuster · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…