2020

Stable Neural Flows

Massaroli, Stefano, Poli, Michael, Bin, Michelangelo et al.

Understand

We introduce a provably stable variant of neural ordinary differential equations (neural ODEs) whose trajectories evolve on an energy functional parametrised by a neural network.

Stable neural flows provide an implicit guarantee on asymptotic stability of the depth-flows, leading to robustness against input perturbations and low computational burden for the numerical solver.
The learning procedure is cast as an optimal control problem, and an approximate solution is proposed based on adjoint sensivity analysis.
We further introduce novel regularizers designed to ease the optimization process and speed up convergence.

Built on

The mathematical theory of optimal processes
Lev Semenovich Pontryagin, EF Mishchenko, VG Boltyanskii, and RV Gamkrelidze · 1962
Earlier work this paper cites.
Minimum cross-entropy pattern classification and cluster analysis
John E Shore and Robert M Gray · 1982
Earlier work this paper cites.
The adjoint process in stochastic optimal control
Robert J Elliott and Michael Kohlmann · 1989
Earlier work this paper cites.
Putting energy back in control
Romeo Ortega, Arjan J Van Der Schaft, Iven Mareels, and Bernhard Maschke · 2001
Earlier work this paper cites.
Adjoint sensitivity analysis for differential-algebraic equations: algorithms and software
Yang Cao, Shengtai Li, and Linda Petzold · 2002
Earlier work this paper cites.
Double continuum limit of deep neural networks
Sho Sonoda and Noboru Murata · 2017
Earlier work this paper cites.
Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations
Original
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong · 2017
Earlier work this paper cites.
Stable architectures for deep neural networks
Eldad Haber and Lars Ruthotto · 2017
Earlier work this paper cites.

Similar

Neural ordinary differential equations
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud · 2018
Cited alongside, same era.
Transport analysis of infinitely deep neural network
Sho Sonoda and Noboru Murata · 2019
Cited alongside, same era.
Latent ordinary differential equations for irregularly-sampled time series
Yulia Rubanova, Tian Qi Chen, and David K Duvenaud · 2019
Cited alongside, same era.
Ode2vae: Deep generative second order odes with bayesian neural networks
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki · 2019
Cited alongside, same era.
Neural jump stochastic differential equations
Junteng Jia and Austin R Benson · 2019
Cited alongside, same era.
Graph neural ordinary differential equations
Original
Michael Poli, Stefano Massaroli, Junyoung Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park · 2019
Cited alongside, same era.
Hamiltonian neural networks
Samuel Greydanus, Misko Dzamba, and Jason Yosinski · 2019
Cited alongside, same era.
Antisymmetricrnn: A dynamical system view on recurrent neural networks
Original
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi · 2019
Cited alongside, same era.

Then

On robustness of neural ordinary differential equations
YAN Hanshu, DU Jiawei, TAN Vincent, and FENG Jiashi · 2019
Later among the works it cites.
Augmented neural odes
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh · 2019
Later among the works it cites.
Port–hamiltonian approach to neural network training
S. Massaroli, M. Poli, F. Califano, A. Faragasso, J. Park, A. Yamashita, and H. Asama · 2019
Later among the works it cites.
Gradient flow algorithms for density propagation in stochastic systems
Kenneth Caluya and Abhishek Halder · 2019
Later among the works it cites.
A survey of optimization methods from a machine learning perspective
Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhao · 2019
Later among the works it cites.
Scalable gradients for stochastic differential equations
Original
Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud · 2020
Closest in time.
Universal differential equations for scientific machine learning
Original
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, and Ali Ramadhan · 2020
Closest in time.
Dissecting neural odes
Original
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…