2020

Stochasticity in Neural ODEs: An Empirical Study

Oganesyan, Viktor, Volokhova, Alexandra, Vetrov, Dmitry

Understand

Stochastic regularization of neural networks (e.g.

dropout) is a wide-spread technique in deep learning that allows for better generalization.
Despite its success, continuous-time models, such as neural ordinary differential equation (ODE), usually rely on a completely deterministic feed-forward operation.
This work provides an empirical study of stochastically regularized neural ODE on several image-classification tasks (CIFAR-10, CIFAR-100, TinyImageNet).

Learning multiple layers of features from tiny images
Alex Krizhevsky, Geoffrey Hinton, et al · 2009
Earlier work this paper cites.
Machine learning: a probabilistic perspective
Kevin P Murphy · 2012
Earlier work this paper cites.
Dropout: a simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
Sergey Ioffe and Christian Szegedy · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Automatic differentiation in pytorch
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer · 2017
Earlier work this paper cites.

Neural ordinary differential equations
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud · 2018
Cited alongside, same era.
Ffjord: Free-form continuous dynamics for scalable reversible generative models
Original
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud · 2018
Cited alongside, same era.
Latent ordinary differential equations for irregularly-sampled time series
Yulia Rubanova, Tian Qi Chen, and David K Duvenaud · 2019
Cited alongside, same era.
Ode2vae: Deep generative second order odes with bayesian neural networks
Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki · 2019
Cited alongside, same era.
Neural jump stochastic differential equations
Junteng Jia and Austin R Benson · 2019
Cited alongside, same era.
Theoretical guarantees for sampling and inference in generative models with latent diffusions
Original
Belinda Tzen and Maxim Raginsky · 2019
Cited alongside, same era.
https://tiny-imagenet.herokuapp.com
Cited in the paper.

Neural sde: Stabilizing neural ode networks with stochastic noise
Original
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh · 2019
Later among the works it cites.
Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit
Original
Belinda Tzen and Maxim Raginsky · 2019
Later among the works it cites.
Anode: Unconditionally accurate memory-efficient gradients for neural odes
Original
Amir Gholami, Kurt Keutzer, and George Biros · 2019
Later among the works it cites.
Augmented neural odes
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh · 2019
Later among the works it cites.
Scalable gradients for stochastic differential equations
Original
Xuechen Li, Ting-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud · 2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…