Towards Understanding Normalization in Neural ODEs

Theory of ordinary differential equations

Earl A Coddington and Norman Levinson · 1955

Earlier work this paper cites.

Long short-term memory

Sepp Hochreiter and Jürgen Schmidhuber · 1997

Earlier work this paper cites.

Improving predictive inference under covariate shift by weighting the log-likelihood function

Hidetoshi Shimodaira · 2000

Earlier work this paper cites.

Batch normalization: Accelerating deep network training by reducing internal covariate shift

Original

Sergey Ioffe and Christian Szegedy · 2015

Earlier work this paper cites.

Layer normalization

Original

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016

Earlier work this paper cites.

Recurrent batch normalization

Original

Tim Cooijmans, Nicolas Ballas, César Laurent, Çağlar Gülçehre, and Aaron Courville · 2016

Cited alongside, same era.

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016

Cited alongside, same era.

Weight normalization: A simple reparameterization to accelerate training of deep neural networks

Tim Salimans and Durk P Kingma · 2016

Cited alongside, same era.

Neural ordinary differential equations

Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud · 2018

Cited alongside, same era.

Differentiable learning-to-normalize via switchable normalization

Original

Ping Luo, Jiamin Ren, Zhanglin Peng, Ruimao Zhang, and Jingyu Li · 2018

Cited alongside, same era.

Spectral normalization for generative adversarial networks

Original

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida · 2018

Later among the works it cites.

How does batch normalization help optimization?

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry · 2018

Later among the works it cites.

Anode: Unconditionally accurate memory-efficient gradients for neural odes

Original

Amir Gholami, Kurt Keutzer, and George Biros · 2019

Later among the works it cites.

ANODEV2: A coupled neural ODE evolution framework

Original

Tianjun Zhang, Zhewei Yao, Amir Gholami, Kurt Keutzer, Joseph Gonzalez, George Biros, and Michael Mahoney · 2019

Later among the works it cites.