2015

A Probabilistic Theory of Deep Learning

Patel, Ankit B., Nguyen, Tan, Baraniuk, Richard G.

Understand

A grand challenge in machine learning is the development of computational algorithms that match or outperform humans in perceptual inference tasks that are complicated by nuisance variation.

For instance, visual object recognition involves the unknown object position, orientation, and scale in object recognition while speech recognition involves the unknown voice pronunciation, pitch, and speed.
Recently, a new breed of deep learning algorithms have emerged for high-nuisance inference tasks that routinely yield pattern recognition systems with near- or super-human capabilities.
But a fundamental question remains: Why do they work? Intuitions abound, but a coherent framework for understanding, analyzing, and synthesizing deep learning architectures has remained elusive.

Built on

J. Pearl, “Probabilistic reasoning in intelligent systems: Networks of plausible inference. morgan kauffman pub,” 1988
1988
Earlier work this paper cites.
T. Vámos, “Judea pearl: Probabilistic reasoning in intelligent systems,” Decision Support Systems , vol. 8, no. 1, pp. 73–75, 1992
1992
Earlier work this paper cites.
H. Schmid, “Part-of-speech tagging with neural networks,” in Proceedings of the 15th Conference on Computational Linguistics - Volume 1 , ser. COLING ’94. Stroudsburg, PA, USA: Association for Computational Linguistics, 1994, pp. 172–176. [Online]. Available: http://dx.doi.org/10.3115/991886.991915
1994
Earlier work this paper cites.
J. H. Searcy and J. C. Bartlett, “Inversion and processing of component and spatial-relational information in faces.” Journal of experimental psychology. Human perception and performance , vol. 22, no. 4, pp. 904–915, Aug. 1996
1996
Earlier work this paper cites.
Z. Ghahramani, G. E. Hinton et al. , “The em algorithm for mixtures of factor analyzers,” Technical Report CRG-TR-96-1, University of Toronto, Tech. Rep., 1996
1996
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
M. Jordan, Learning in Graphical Models , ser. Adaptive computation and machine learning. London, 1998. [Online]. Available: https://books.google.com/books?id=zac7L4LbNtUC
1998
Earlier work this paper cites.
M. I. Jordan and T. J. Sejnowski, Graphical models: Foundations of neural computation . MIT press, 2001
2001
Earlier work this paper cites.
F. R. Kschischang, B. J. Frey, and H.-A. Loeliger, “Factor graphs and the sum-product algorithm,” Information Theory, IEEE Transactions on , vol. 47, no. 2, pp. 498–519, 2001
2001
Earlier work this paper cites.
S. Roweis and Z. Ghahramani, “Learning nonlinear dynamical systems using the expectation–maximization algorithm,” Kalman filtering and neural networks , p. 175, 2001
2001
Earlier work this paper cites.
B. M. Wilamowski, S. Iplikci, O. Kaynak, and M. Ö. Efe, “An algorithm for fast convergence in training neural networks,” in Proceedings of the international joint conference on neural networks , vol. 2, 2001, pp. 1778–1782
2001
Earlier work this paper cites.
L. Breiman, “Random forests,” Machine learning , vol. 45, no. 1, pp. 5–32, 2001
2001
Earlier work this paper cites.
A. Jordan, “On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes,” Advances in neural information processing systems , vol. 14, p. 841, 2002
2002
Earlier work this paper cites.
R. Hartley and A. Zisserman, Multiple view geometry in computer vision . Cambridge university press, 2003
2003
Earlier work this paper cites.
D. Griffiths and M. Tenenbaum, “Hierarchical topic models and the nested chinese restaurant process,” Advances in neural information processing systems , vol. 16, p. 17, 2004
2004
Earlier work this paper cites.
A. Hyvärinen, J. Karhunen, and E. Oja, Independent component analysis . John Wiley & Sons, 2004, vol. 46
2004
Earlier work this paper cites.
P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient belief propagation for early vision,” International journal of computer vision , vol. 70, no. 1, pp. 41–54, 2006
2006
Earlier work this paper cites.
C. M. Bishop et al. , Pattern recognition and machine learning . springer New York, 2006, vol. 4, no. 4
2006
Earlier work this paper cites.
L. Wiskott, “How does our visual system achieve shift and size invariance,” JL van Hemmen and TJ Sejnowski, editors , vol. 23, pp. 322–340, 2006
2006
Earlier work this paper cites.
C. M. Bishop, J. Lasserre et al. , “Generative or discriminative? getting the best of both worlds,” Bayesian Statistics , vol. 8, pp. 3–24, 2007
2007
Earlier work this paper cites.

Similar

F. Anselmi, J. Mutch, and T. Poggio, “Magic Materials,” Proceedings of the National Academy of Sciences , vol. 104, no. 51, pp. 20 167–20 172, Dec. 2007
2007
Cited alongside, same era.
X. Miao and R. P. Rao, “Learning the lie groups of visual invariance,” Neural computation , vol. 19, no. 10, pp. 2665–2693, 2007
2007
Cited alongside, same era.
O. Cappé and E. Moulines, “Online em algorithm for latent data models,” Journal of the Royal Statistical Society , 2008
2008
Cited alongside, same era.
N. Pinto, D. D. Cox, and J. J. DiCarlo, “Why is Real-World Visual Object Recognition Hard?” PLoS Computational Biology , vol. 4, no. 1, p. e27, 2008
2008
Cited alongside, same era.
K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, “What is the best multi-stage architecture for object recognition?” in Computer Vision, 2009 IEEE 12th International Conference on . IEEE, 2009, pp. 2146–2153
2009
Cited alongside, same era.
N. Kumar, S. Satoor, and I. Buck, “Fast parallel expectation maximization for gaussian mixture models on gpus using cuda,” in High Performance Computing and Communications, 2009. HPCC’09. 11th IEEE International Conference on . IEEE, 2009, pp. 103–109
2009
Cited alongside, same era.
J. Sohl-Dickstein, J. C. Wang, and B. A. Olshausen, “An unsupervised algorithm for learning lie group transformations,” arXiv preprint arXiv:1001.1027 , 2010
Original
2010
Cited alongside, same era.
J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman, “How to grow a mind: Statistics, structure, and abstraction,” science , vol. 331, no. 6022, pp. 1279–1285, 2011
2011
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, and G. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” NIPS , pp. 1–9, Nov. 2012
2012
Cited alongside, same era.
J. Lücke and A.-S. Sheikh, “Closed-form em for sparse coding and its application to source separation,” in Latent Variable Analysis and Signal Separation . Springer, 2012, pp. 213–221
2012
Cited alongside, same era.
I. Goodfellow, A. Courville, and Y. Bengio, “Large-scale feature learning with spike-and-slab sparse coding,” arXiv preprint arXiv:1206.6407 , 2012
Original
2012
Cited alongside, same era.
Y. Tang, R. Salakhutdinov, and G. Hinton, “Deep mixtures of factor analysers,” arXiv preprint arXiv:1206.4635 , 2012
Original
2012
Cited alongside, same era.

Then

K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” arXiv preprint arXiv:1312.6034 , 2013
Original
2013
Later among the works it cites.
J. Bruna and S. Mallat, “Invariant scattering convolution networks,” Pattern Analysis and Machine Intelligence, IEEE Transactions on , vol. 35, no. 8, pp. 1872–1886, 2013
2013
Later among the works it cites.
S. Arora, A. Bhaskara, R. Ge, and T. Ma, “Provable bounds for learning some deep representations,” arXiv preprint arXiv:1310.6343 , 2013
Original
2013
Later among the works it cites.
F. Anselmi, J. Z. Leibo, L. Rosasco, J. Mutch, A. Tacchetti, and T. Poggio, “Unsupervised learning of invariant representations in hierarchical architectures,” arXiv preprint arXiv:1311.4158 , 2013
Original
2013
Later among the works it cites.
A. Graves, A.-R. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on . IEEE, 2013, pp. 6645–6649
2013
Later among the works it cites.
A. Graves, N. Jaitly, and A.-R. Mohamed, “Hybrid speech recognition with deep bidirectional lstm,” in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on . IEEE, 2013, pp. 273–278
2013
Later among the works it cites.
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in Computer Vision–ECCV 2014 . Springer, 2014, pp. 818–833
2014
Later among the works it cites.
A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates et al. , “Deepspeech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.5567 , 2014
Original
2014
Later among the works it cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” arXiv preprint arXiv:1409.4842 , 2014
Original
2014
Later among the works it cites.
Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to human-level performance in face verification,” in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on . IEEE, 2014, pp. 1701–1708
2014
Later among the works it cites.
A. van den Oord and B. Schrauwen, “Factoring variations in natural images with deep gaussian mixture models,” in Advances in Neural Information Processing Systems , 2014, pp. 3518–3526
2014
Later among the works it cites.
G. Hinton, “What’s wrong with convolutional nets?” 2014, available from the MIT TechTV website
2014
Later among the works it cites.
D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo, “Performance-optimized hierarchical models predict neural responses in higher visual cortex,” Proceedings of the National Academy of Sciences , vol. 111, no. 23, pp. 8619–8624, 2014
2014
Later among the works it cites.
P. Mehta and D. J. Schwab, “An exact mapping between the variational renormalization group and deep learning,” arXiv preprint arXiv:1410.3831 , 2014
Original
2014
Later among the works it cites.
V. Michalski, R. Memisevic, and K. Konda, “Modeling sequential data using higher-order relational features and predictive training,” arXiv preprint arXiv:1402.2333 , 2014
Original
2014
Later among the works it cites.
J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks , vol. 61, pp. 85–117, 2015
2015
Closest in time.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167 , 2015
Original
2015
Closest in time.
N.-Q. Pham, H.-S. Le, D.-D. Nguyen, and T.-G. Ngo, “A study of feature combination in gesture recognition with kinect,” in Knowledge and Systems Engineering . Springer, 2015, pp. 459–471
2015
Closest in time.
F. Anselmi, L. Rosasco, and T. Poggio, “On invariance and selectivity in representation learning,” arXiv preprint arXiv:1503.05938 , 2015
Original
2015
Closest in time.
F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” arXiv preprint arXiv:1503.03832 , 2015
Original
2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…