2020

STEER: Simple Temporal Regularization For Neural ODEs

Ghosh, Arnab, Behl, Harkirat Singh, Dupont, Emilien et al.

Understand

Training Neural Ordinary Differential Equations (ODEs) is often computationally expensive.

Indeed, computing the forward pass of such models involves solving an ODE which can become arbitrarily complex during training.
Recent works have shown that regularizing the dynamics of the ODE can partially alleviate this.
In this paper we propose a new regularization technique: randomly sampling the end time of the ODE during training.

Built on

Neural odes as the deep limit of resnets with constant weights
Original
Avelin, B. and Nyström, K. (2019) · 1906
Earlier work this paper cites.
Neural sde: Stabilizing neural ode networks with stochastic noise
Original
Liu, X., Xiao, T., Si, S., Cao, Q., Kumar, S., and Hsieh, C.-J. (2019) · 1906
Earlier work this paper cites.
Approximation capabilities of neural ordinary differential equations
Original
Zhang, H., Gao, X., Unterman, J., and Arodz, T. (2019) · 1907
Earlier work this paper cites.
Deep neural networks, generic universal interpolation, and controlled odes
Original
Cuchiero, C., Larsson, M., and Teichmann, J. (2019) · 1908
Earlier work this paper cites.
Mean-field neural odes via relaxed optimal control
Original
Jabir, J.-F., Šiška, D., and Szpruch, Ł. (2019) · 1912
Earlier work this paper cites.
Normalizing flows for probabilistic modeling and inference
Original
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2019) · 1912
Earlier work this paper cites.
Continuous graph neural networks
Original
Xhonneux, L.-P. A., Qu, M., and Tang, J. (2019) · 1912
Earlier work this paper cites.
Integration of stiff equations
Curtiss, C. F. and Hirschfelder, J. O. (1952) · 1952
Earlier work this paper cites.
Error analysis for a class of methods for stiff non-linear initial value problems
Dahlquist, G. (1976) · 1976
Earlier work this paper cites.
A family of embedded runge-kutta formulae
Dormand, J. R. and Prince, P. J. (1980) · 1980
Earlier work this paper cites.
33 years of numerical instability, part i
Dahlquist, G. (1985) · 1985
Earlier work this paper cites.
Handwritten digit recognition with a back-propagation network
LeCun, Y., Boser, B. E., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W. E., and Jackel, L. D. (1990) · 1990
Earlier work this paper cites.
Gradient calculations for dynamic recurrent neural networks: A survey
Pearlmutter, B. A. (1995) · 1995
Earlier work this paper cites.
Solving ordinary differential equations II
Wanner, G. and Hairer, E. (1996) · 1996
Earlier work this paper cites.
Artificial neural networks for solving ordinary and partial differential equations
Lagaris, I. E., Likas, A., and Fotiadis, D. I. (1998) · 1998
Earlier work this paper cites.
Scalable gradients for stochastic differential equations
Original
Li, X., Wong, T.-K. L., Chen, R. T., and Duvenaud, D. (2020a) · 2001
Earlier work this paper cites.
Stochastic normalizing flows
Original
Hodgkinson, L., van der Heide, C., Roosta, F., and Mahoney, M. W. (2020) · 2002
Earlier work this paper cites.
Dissecting neural odes
Original
Massaroli, S., Poli, M., Park, J., Yamashita, A., and Asama, H. (2020b) · 2002
Earlier work this paper cites.
Stochasticity in neural odes: An empirical study
Original
Oganesyan, V., Volokhova, A., and Vetrov, D. (2020) · 2002
Earlier work this paper cites.
Stochastic normalizing flows
Original
Wu, H., Köhler, J., and Noé, F. (2020) · 2002
Earlier work this paper cites.

Similar

Stable neural flows
Original
Massaroli, S., Poli, M., Bin, M., Park, J., Yamashita, A., and Asama, H. (2020a) · 2003
Cited alongside, same era.
Time dependence in non-autonomous neural odes
Original
Davis, J. Q., Choromanski, K., Varley, J., Lee, H., Slotine, J.-J., Likhosterov, V., Weller, A., Makadia, A., and Sindhwani, V. (2020) · 2005
Cited alongside, same era.
Model-based reinforcement learning for semi-markov decision processes with neural odes
Original
Du, J., Futoma, J., and Doshi-Velez, F. (2020) · 2006
Cited alongside, same era.
On second order behaviour in augmented neural odes
Original
Norcliffe, A., Bodnar, C., Day, B., Simidjievski, N., and Liò, P. (2020) · 2006
Cited alongside, same era.
Learning differential equations that are easy to solve
Original
Kelly, J., Bettencourt, J., Johnson, M. J., and Duvenaud, D. (2020) · 2007
Cited alongside, same era.
On early stopping in gradient descent learning
Yao, Y., Rosasco, L., and Caponnetto, A. (2007) · 2007
Cited alongside, same era.
Exchangeable neural ode for set modeling
Original
Li, Y., Yi, H., Bender, C. M., Shan, S., and Oliva, J. B. (2020b) · 2008
Cited alongside, same era.
ImageNet: A Large-Scale Hierarchical Image Database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009) · 2009
Cited alongside, same era.
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., et al. (2009) · 2009
Cited alongside, same era.
Numerical methods for engineers
Chapra, S. C., Canale, R. P., et al. (2010) · 2010
Cited alongside, same era.
Fifty years of stiffness
Brugnano, L., Mazzia, F., and Trigiante, D. (2011) · 2011
Cited alongside, same era.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and N, A. Y. (2011) · 2011
Cited alongside, same era.

Then

Residual networks behave like ensembles of relatively shallow networks
Veit, A., Wilber, M. J., and Belongie, S. (2016) · 2016
Later among the works it cites.
Stable architectures for deep neural networks
Haber, E. and Ruthotto, L. (2017) · 2017
Later among the works it cites.
Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations
Original
Lu, Y., Zhong, A., Li, Q., and Dong, B. (2017) · 2017
Later among the works it cites.
Masked autoregressive flow for density estimation
Papamakarios, G., Pavlakou, T., and Murray, I. (2017) · 2017
Later among the works it cites.
Differentialequations. jl–a performant and feature-rich ecosystem for solving differential equations in julia
Rackauckas, C. and Nie, Q. (2017) · 2017
Later among the works it cites.
A proposal on machine learning via dynamical systems
Weinan, E. (2017) · 2017
Later among the works it cites.
Neural ordinary differential equations
Chen, T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018) · 2018
Later among the works it cites.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
Grathwohl, W., Chen, R. T., Bettencourt, J., Sutskever, I., and Duvenaud, D. (2018) · 2018
Later among the works it cites.
Glow: Generative flow with invertible 1x1 convolutions
Kingma, D. P. and Dhariwal, P. (2018) · 2018
Later among the works it cites.
Augmented neural odes
Dupont, E., Doucet, A., and Teh, Y. W. (2019) · 2019
Later among the works it cites.
Interactive sketch & fill: Multiclass sketch-to-image translation
Ghosh, A., Zhang, R., Dokania, P. K., Wang, O., Efros, A. A., Torr, P. H., and Shechtman, E. (2019) · 2019
Later among the works it cites.
On robustness of neural ordinary differential equations
Hanshu, Y., Jiawei, D., Vincent, T., and Jiashi, F. (2019) · 2019
Later among the works it cites.
Neural jump stochastic differential equations
Jia, J. and Benson, A. R. (2019) · 2019
Later among the works it cites.
Simple video generation using neural odes
Kanaa, D., Voleti, V., Kahou, S., and Pal, C. (2019) · 2019
Later among the works it cites.
Latent ordinary differential equations for irregularly-sampled time series
Rubanova, Y., Chen, T. Q., and Duvenaud, D. K. (2019) · 2019
Later among the works it cites.
Deep neural networks motivated by partial differential equations
Ruthotto, L. and Haber, E. (2019) · 2019
Later among the works it cites.
Resnets ensemble via the feynman-kac formalism to improve natural and robust accuracies
Wang, B., Shi, Z., and Osher, S. (2019) · 2019
Later among the works it cites.
Ode2vae: Deep generative second order odes with bayesian neural networks
Yildiz, C., Heinonen, M., and Lahdesmaki, H. (2019) · 2019
Later among the works it cites.
How to train your neural ode
Finlay, C., Jacobsen, J.-H., Nurbekyan, L., and Oberman, A. M. (2020) · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…