2019

Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective

Liu, Guan-Horng, Theodorou, Evangelos A.

Understand

Attempts from different disciplines to provide a fundamental understanding of deep learning have advanced rapidly in recent years, yet a unified framework remains relatively limited.

In this article, we provide one possible way to align existing branches of deep learning theory through the lens of dynamical system and optimal control.
By viewing deep neural networks as discrete-time nonlinear dynamical systems, we can analyze how information propagates through layers using mean field theory.
When optimization algorithms are further recast as controllers, the ultimate goal of training processes can be formulated as an optimal control problem.

Built on

——, “Where is the information in a deep neural network?” CoRR , vol. abs/1905.12213, 2019. [Online]. Available: http://arxiv.org/abs/1905.12213
Original
1905
Earlier work this paper cites.
A. Kolmogoroff, “Über die analytischen methoden in der wahrscheinlichkeitsrechnung,” Mathematische Annalen , vol. 104, no. 1, pp. 415–458, 1931
1931
Earlier work this paper cites.
K. Itô, On stochastic differential equations . American Mathematical Soc., 1951, vol. 4
1951
Earlier work this paper cites.
V. G. Boltyanskii, R. V. Gamkrelidze, and L. S. Pontryagin, “The theory of optimal processes. i. the maximum principle,” TRW SPACE TECHNOLOGY LABS LOS ANGELES CALIF, Tech. Rep., 1960
1960
Earlier work this paper cites.
R. E. Bellman and R. E. Kalaba, Selected papers on mathematical trends in control theory . Dover Publications, 1964
1964
Earlier work this paper cites.
Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, “Handwritten digit recognition with a back-propagation network,” in Advances in neural information processing systems , 1990, pp. 396–404
1990
Earlier work this paper cites.
G. F. Franklin, J. D. Powell, A. Emami-Naeini, and J. D. Powell, Feedback control of dynamic systems . Addison-Wesley Reading, MA, 1994, vol. 3
1994
Earlier work this paper cites.
G. N. Milstein, Numerical integration of stochastic differential equations . Springer Science & Business Media, 1994, vol. 313
1994
Earlier work this paper cites.
D. P. Bertsekas, D. P. Bertsekas, D. P. Bertsekas, and D. P. Bertsekas, Dynamic programming and optimal control . Athena scientific Belmont, MA, 1995, vol. 1, no. 2
1995
Earlier work this paper cites.
C. K. Williams, “Computing with infinite networks,” in Advances in neural information processing systems , 1997, pp. 295–301
1997
Earlier work this paper cites.
R. Jordan, D. Kinderlehrer, and F. Otto, “The variational formulation of the fokker–planck equation,” SIAM journal on mathematical analysis , vol. 29, no. 1, pp. 1–17, 1998
1998
Earlier work this paper cites.
N. Qian, “On the momentum term in gradient descent learning algorithms,” Neural networks , vol. 12, no. 1, pp. 145–151, 1999
1999
Earlier work this paper cites.
S. P. Coraluppi and S. I. Marcus, “Risk-sensitive and minimax control of discrete-time, finite-state markov decision processes,” Automatica , vol. 35, no. 2, pp. 301–309, 1999
1999
Earlier work this paper cites.
B. Øksendal, “Stochastic differential equations,” in Stochastic differential equations . Springer, 2003, pp. 65–84
2003
Earlier work this paper cites.
A. Bovier, M. Eckhoff, V. Gayrard, and M. Klein, “Metastability in reversible diffusion processes i: Sharp asymptotics for capacities and exit times,” Journal of the European Mathematical Society , vol. 6, no. 4, pp. 399–424, 2004
2004
Earlier work this paper cites.
L. Ambrosio, N. Gigli, and G. Savaré, Gradient flows: in metric spaces and in the space of probability measures . Springer Science & Business Media, 2008
2008
Earlier work this paper cites.
T. Başar and P. Bernhard, H-infinity optimal control and related minimax design problems: a dynamic game approach . Springer Science & Business Media, 2008
2008
Earlier work this paper cites.
X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics , 2010, pp. 249–256
2010
Earlier work this paper cites.
E. Moulines and F. R. Bach, “Non-asymptotic analysis of stochastic approximation algorithms for machine learning,” in Advances in Neural Information Processing Systems , 2011, pp. 451–459
2011
Earlier work this paper cites.
W. Xu, “Towards optimal one pass large scale learning with averaged stochastic gradient descent,” arXiv preprint arXiv:1107.2490 , 2011
Original
2011
Earlier work this paper cites.
M. Welling and Y. W. Teh, “Bayesian learning via stochastic gradient langevin dynamics,” in Proceedings of the 28th international conference on machine learning (ICML-11) , 2011, pp. 681–688
2011
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
N. Srivastava and R. R. Salakhutdinov, “Multimodal learning with deep boltzmann machines,” in Advances in neural information processing systems , 2012, pp. 2222–2230
2012
Earlier work this paper cites.
K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034 , 2013
Original
2013
Earlier work this paper cites.
I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in International conference on machine learning , 2013, pp. 1139–1147
2013
Earlier work this paper cites.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199 , 2013
Original
2013
Earlier work this paper cites.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
Original
2014
Earlier work this paper cites.
Y. N. Dauphin, R. Pascanu, C. Gulcehre, K. Cho, S. Ganguli, and Y. Bengio, “Identifying and attacking the saddle point problem in high-dimensional non-convex optimization,” in Advances in neural information processing systems , 2014, pp. 2933–2941
2014
Earlier work this paper cites.
G. A. Pavliotis, Stochastic processes and applications: diffusion processes, the Fokker-Planck and Langevin equations . Springer, 2014, vol. 60
2014
Earlier work this paper cites.
M. Hardt, B. Recht, and Y. Singer, “Train faster, generalize better: Stability of stochastic gradient descent,” arXiv preprint arXiv:1509.01240 , 2015
Original
2015
Earlier work this paper cites.
N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in 2015 IEEE Information Theory Workshop (ITW) . IEEE, 2015, pp. 1–5
2015
Earlier work this paper cites.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al. , “Mastering the game of go with deep neural networks and tree search,” nature , vol. 529, no. 7587, p. 484, 2016
2016
Earlier work this paper cites.
S. Wang, W. Liu, J. Wu, L. Cao, Q. Meng, and P. J. Kennedy, “Training deep neural networks on imbalanced data sets,” in 2016 international joint conference on neural networks (IJCNN) . IEEE, 2016, pp. 4368–4374
2016
Earlier work this paper cites.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, “Exponential expressivity in deep neural networks through transient chaos,” in Advances in neural information processing systems , 2016, pp. 3360–3368
2016
Earlier work this paper cites.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” arXiv preprint arXiv:1611.03530 , 2016
Original
2016
Earlier work this paper cites.
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” arXiv preprint arXiv:1609.04836 , 2016
Original
2016
Earlier work this paper cites.

Similar

S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, “Deep information propagation,” arXiv preprint arXiv:1611.01232 , 2016
Original
2016
Cited alongside, same era.
P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina, “Entropy-sgd: Biasing gradient descent into wide valleys,” arXiv preprint arXiv:1611.01838 , 2016
Original
2016
Cited alongside, same era.
Y. W. Teh, A. H. Thiery, and S. J. Vollmer, “Consistency and fluctuations for stochastic gradient langevin dynamics,” The Journal of Machine Learning Research , vol. 17, no. 1, pp. 193–225, 2016
2016
Cited alongside, same era.
C. Li, C. Chen, D. Carlson, and L. Carin, “Preconditioned stochastic gradient langevin dynamics for deep neural networks,” in Thirtieth AAAI Conference on Artificial Intelligence , 2016
2016
Cited alongside, same era.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
Cited alongside, same era.
M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu, “Reinforcement learning with unsupervised auxiliary tasks,” arXiv preprint arXiv:1611.05397 , 2016
Original
2016
Cited alongside, same era.
Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, “Rl 2 : Fast reinforcement learning via slow reinforcement learning,” arXiv preprint arXiv:1611.02779 , 2016
Original
2016
Cited alongside, same era.
J. D. Lee, I. Panageas, G. Piliouras, M. Simchowitz, M. I. Jordan, and B. Recht, “First-order methods almost always avoid saddle points,” arXiv preprint arXiv:1710.07406 , 2017
Original
2017
Cited alongside, same era.
R. Shwartz-Ziv and N. Tishby, “Opening the black box of deep neural networks via information,” arXiv preprint arXiv:1703.00810 , 2017
Original
2017
Cited alongside, same era.
G. Yang and S. Schoenholz, “Mean field residual networks: On the edge of chaos,” in Advances in neural information processing systems , 2017, pp. 7103–7114
2017
Cited alongside, same era.
J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein, “Deep neural networks as gaussian processes,” arXiv preprint arXiv:1711.00165 , 2017
Original
2017
Cited alongside, same era.
B. Neyshabur, R. Tomioka, R. Salakhutdinov, and N. Srebro, “Geometry of optimization and implicit regularization in deep learning,” arXiv preprint arXiv:1705.03071 , 2017
Original
2017
Cited alongside, same era.

Then

R. Karakida, S. Akaho, and S.-i. Amari, “Universal statistics of fisher information in deep neural networks: Mean field approach,” arXiv preprint arXiv:1806.01316 , 2018
Original
2018
Later among the works it cites.
J. Pennington, S. S. Schoenholz, and S. Ganguli, “The emergence of spectral universality in deep networks,” arXiv preprint arXiv:1802.09979 , 2018
Original
2018
Later among the works it cites.
A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani, “Gaussian process behaviour in wide deep neural networks,” arXiv preprint arXiv:1804.11271 , 2018
Original
2018
Later among the works it cites.
A. Garriga-Alonso, L. Aitchison, and C. E. Rasmussen, “Deep convolutional networks as shallow gaussian processes,” arXiv preprint arXiv:1808.05587 , 2018
Original
2018
Later among the works it cites.
D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro, “The implicit bias of gradient descent on separable data,” The Journal of Machine Learning Research , vol. 19, no. 1, pp. 2822–2878, 2018
2018
Later among the works it cites.
S. Gunasekar, J. Lee, D. Soudry, and N. Srebro, “Characterizing implicit bias in terms of optimization geometry,” arXiv preprint arXiv:1802.08246 , 2018
Original
2018
Later among the works it cites.
S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro, “Implicit bias of gradient descent on linear convolutional networks,” in Advances in Neural Information Processing Systems , 2018, pp. 9482–9491
2018
Later among the works it cites.
Z. Ji and M. Telgarsky, “Gradient descent aligns the layers of deep linear networks,” arXiv preprint arXiv:1810.02032 , 2018
Original
2018
Later among the works it cites.
Z. Allen-Zhu, Y. Li, and Y. Liang, “Learning and generalization in overparameterized neural networks, going beyond two layers,” arXiv preprint arXiv:1811.04918 , 2018
Original
2018
Later among the works it cites.
L. Wu, C. Ma, and E. Weinan, “How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective,” in Advances in Neural Information Processing Systems , 2018, pp. 8279–8288
2018
Later among the works it cites.
R. Kleinberg, Y. Li, and Y. Yuan, “An alternative view: When does sgd escape local minima?” arXiv preprint arXiv:1802.06175 , 2018
Original
2018
Later among the works it cites.
P. Chaudhari, A. Oberman, S. Osher, S. Soatto, and G. Carlier, “Deep relaxation: partial differential equations for optimizing deep neural networks,” Research in the Mathematical Sciences , vol. 5, no. 3, p. 30, 2018
2018
Later among the works it cites.
A. Achille and S. Soatto, “Emergence of invariance and disentanglement in deep representations,” The Journal of Machine Learning Research , vol. 19, no. 1, pp. 1947–1980, 2018
2018
Later among the works it cites.
S. Yaida, “Fluctuation-dissipation relations for stochastic gradient descent,” arXiv preprint arXiv:1810.00004 , 2018
Original
2018
Later among the works it cites.
T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” in Advances in Neural Information Processing Systems , 2018, pp. 6572–6583
2018
Later among the works it cites.
E. Weinan, J. Han, and Q. Li, “A mean-field optimal control formulation of deep learning,” arXiv preprint arXiv:1807.01083 , 2018
Original
2018
Later among the works it cites.
Q. Li and S. Hao, “An optimal control approach to deep learning and applications to discrete-weight neural networks,” arXiv preprint arXiv:1803.01299 , 2018
Original
2018
Later among the works it cites.
J. An, J. Lu, and L. Ying, “Stochastic modified equations for the asynchronous stochastic gradient descent,” arXiv preprint arXiv:1805.08244 , 2018
Original
2018
Later among the works it cites.
Y. Li and Y. Liang, “Learning overparameterized neural networks via stochastic gradient descent on structured data,” in Advances in Neural Information Processing Systems , 2018, pp. 8157–8166
2018
Later among the works it cites.
A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,” in Advances in neural information processing systems , 2018, pp. 8571–8580
2018
Later among the works it cites.
I. Goodfellow, “Defense against the dark arts: An overview of adversarial example security research and future research directions,” arXiv preprint arXiv:1806.04169 , 2018
Original
2018
Later among the works it cites.
L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry, “Adversarially robust generalization requires more data,” in Advances in Neural Information Processing Systems , 2018, pp. 5014–5026
2018
Later among the works it cites.
E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths, “Recasting gradient-based meta-learning as hierarchical bayes,” arXiv preprint arXiv:1801.08930 , 2018
Original
2018
Later among the works it cites.
D. Gilboa, B. Chang, M. Chen, G. Yang, S. S. Schoenholz, E. H. Chi, and J. Pennington, “Dynamical isometry and a mean field theory of lstms and grus,” arXiv preprint arXiv:1901.08987 , 2019
Original
2019
Closest in time.
Z. Zhu, J. Wu, B. Yu, L. Wu, and J. Ma, “The anisotropic noise in stochastic gradient descent: Its behavior of escaping from minima and regularization effects,” in International conference on machine learning , 2019
2019
Closest in time.
A. Achille, M. Rovere, and S. Soatto, “Critical learning periods in deep networks,” in International Conference on Learning Representations , 2019. [Online]. Available: https://openreview.net/forum?id=BkeStsCcKQ
2019
Closest in time.
G. Valle-Perez, C. Q. Camargo, and A. A. Louis, “Deep learning generalizes because the parameter-function map is biased towards simple functions,” in International Conference on Learning Representations , 2019. [Online]. Available: https://openreview.net/forum?id=rye4g3AqFm
2019
Closest in time.
S. Sonoda and N. Murata, “Transport analysis of infinitely deep neural network,” Journal of Machine Learning Research , vol. 20, no. 2, pp. 1–52, 2019. [Online]. Available: http://jmlr.org/papers/v20/16-243.html
2019
Closest in time.
R. T. Q. Chen and D. Duvenaud, “Neural networks with cheap differential operators,” in 2019 ICML Workshop on Invertible Neural Nets and Normalizing Flows (INNF) , 2019
2019
Closest in time.
D. Zhang, T. Zhang, Y. Lu, Z. Zhu, and B. Dong, “You only propagate once: Accelerating adversarial training via maximal principle,” arXiv preprint arXiv:1905.00877 , 2019
Original
2019
Closest in time.
U. Simsekli, L. Sagun, and M. Gurbuzbalaban, “A tail-index analysis of stochastic gradient noise in deep neural networks,” arXiv preprint arXiv:1901.06053 , 2019
Original
2019
Closest in time.
A. Nøkland and L. H. Eidnes, “Training neural networks with local error signals,” arXiv preprint arXiv:1901.06656 , 2019
Original
2019
Closest in time.
G. M. Rotskoff and E. Vanden-Eijnden, “Trainability and accuracy of neural networks: An interacting particle system approach,” arXiv preprint arXiv:1805.00915v3 , 2019
Original
2019
Closest in time.
C. Tao, S. Dai, L. Chen, K. Bai, J. Chen, C. Liu, R. Zhang, G. Bobashev, and L. C. Duke, “Variational annealing of gans: A langevin perspective,” in International Conference on Machine Learning , 2019, pp. 6176–6185
2019
Closest in time.
G. Yang, J. Pennington, V. Rao, J. Sohl-Dickstein, and S. S. Schoenholz, “A mean field theory of batch normalization,” arXiv preprint arXiv:1902.08129 , 2019
Original
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…