2015

Deep Convolutional Networks are Hierarchical Kernel Machines

Anselmi, Fabio, Rosasco, Lorenzo, Tan, Cheston et al.

Understand

In i-theory a typical layer of a hierarchical architecture consists of HW modules pooling the dot products of the inputs to the layer with the transformations of a few templates under a group.

Such layers include as special cases the convolutional layers of Deep Convolutional Networks (DCNs) as well as the non-convolutional layers (when the group contains only the identity).
Rectifying nonlinearities -- which are used by present-day DCNs -- are one of the several nonlinearities admitted by i-theory for the HW module.
We discuss here the equivalence between group averages of linear combinations of rectifying nonlinearities and an associated kernel.

Built on

D. Hubel and T. Wiesel, “Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex,” The Journal of Physiology
1962
Earlier work this paper cites.
San Francisco, CA: Holden-Day, 1965
S. Mikhlin, The problem of the minimum of a quadratic functional · 1965
Earlier work this paper cites.
J. MacQueen, “Some methods of classification and analysis of multivariate observations,” in Proc. 5th Berkeley Symposium on Math., Stat., and Prob
1967
Earlier work this paper cites.
Mathematics in science and engineering, New York, London: Academic Press, 1972
A. E. Albert, Regression and the Moore-Penrose pseudoinverse · 1972
Earlier work this paper cites.
K. Fukushima, “Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position,” Biological Cybernetics
1980
Earlier work this paper cites.
S. Kirkpatrick, C. Gelatt, and M. Vecchi, “Optimization by simulated annealing,” Science
1983
Earlier work this paper cites.
J. Demmel, “The geometry of ill-conditioning,” J. Complexity
1987
Earlier work this paper cites.
D. S. Broomhead and D. Lowe, “Multivariable Functional Interpolation and Adaptive Networks,” Complex Systems 2
1988
Earlier work this paper cites.
T. Poggio and F. Girosi, “A theory of networks for approximation and learning,” Laboratory, Massachusetts Institute of Technology
1989
Earlier work this paper cites.
F. Girosi and T. Poggio, “Representation properties of networks: Kolmogorov’s theorem is irrelevant,” Neural Computation
1989
Earlier work this paper cites.
J. Moody and C. J. Darken, “Fast Learning in Networks of Locally-Tuned Processing Units,” Neural Computation
1989
Earlier work this paper cites.
T. Poggio and F. Girosi, “Regularization algorithms for learning that are equivalent to multilayer networks,” Science
1990
Earlier work this paper cites.

Similar

F. Girosi and T. Poggio, “Networks and the best approximation property,” vol. 63, pp. 169–176, 1990
1990
Cited alongside, same era.
T. Poggio and S. Edelman, “A network that learns to recognize 3D objects,” Nature
1990
Cited alongside, same era.
S. Edelman and T. Poggio, “Bringing the grandmother back into the picture: a memory-based view of object recognition,” a.i. memo 1181, mitai, 1990
1990
Cited alongside, same era.
L. Breiman, “Hinging hyperplanes for regression, classification, and function approximation,” Tech. Rep. 324, Department of Statistics University of California Berkeley, California 94720, 1991
1991
Cited alongside, same era.
J. Mihalik, “Hierarchical vector quantization. of images in transform domain.,” ELEKTROTECHN. CA5, 43, NO. 3. 92,94
1992
Cited alongside, same era.
T. Poggio and F. Girosi, “Extensions of a theory of networks for approximation and learning: dimensionality reduction and clustering,” Laboratory, Massachusetts Institute of Technology
1994
Cited alongside, same era.
F. Girosi, M. Jones, and T. Poggio, “Regularization theory and neural networks architectures,” Neural Computation
1995
Cited alongside, same era.
M. Riesenhuber and T. Poggio, “Hierarchical models of object recognition,” Nature Neuroscience
2000
Cited alongside, same era.
C. A. Micchelli, Y. Xu, and H. Zhang, “Universal kernels.,” Journal of Machine Learning Research
2006
Cited alongside, same era.
T. Serre, A. Oliva, and T. Poggio, “A feedforward architecture accounts for rapid categorization,” Proceedings of the National Academy of Sciences of the United States of America
2007
Cited alongside, same era.
B. Haasdonk and H. Burkhardt, “Invariant kernel functions for pattern analysis and machine learning,” Mach. Learn
2007
Cited alongside, same era.

Then

M. Kouh and T. Poggio, “A canonical neural circuit for cortical nonlinear operations,” Neural computation
2008
Later among the works it cites.
Y. Cho and L. K. Saul, “Kernel methods for deep learning,” in Advances in Neural Information Processing Systems 22
2009
Later among the works it cites.
B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. R. G. Lanckriet, “Hilbert space embeddings and metrics on probability measures,” Journal of Machine Learning Research
2010
Later among the works it cites.
M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks.,” CoRR
Original
2013
Later among the works it cites.
F. Anselmi, J. Z. Leibo, L. Rosasco, J. Mutch, A. Tacchetti, and T. Poggio, “Unsupervised learning of invariant representations in hierarchical architectures,” arXiv preprint arXiv:1311.4158
Original
2013
Later among the works it cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, B. M., A. Berg, and F.-F. L., “Large scale visual recognition challenge,” ImageNet, arXiv:1409.0575
Original
2014
Later among the works it cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” ArXiv e-prints
2014
Later among the works it cites.
N. Cohen and A. Shashua, “Simnets: A generalization of convolutional networks,” CoRR
Original
2014
Later among the works it cites.
D. Yamins, H. Hong, C. Cadieu, E. Solomon, D. Seibert, and J. DiCarlo, “Performance-optimized hierarchical models predict neural responses in higher visual cortex,” Proceedings of the National Academy of Sciences
2014
Later among the works it cites.
F. Anselmi, L. Rosasco, and T. Poggio, “On invariance and selectivity in representation learning,” arXiv:1503.05938 and CBMM memo n 29
Original
2015
Closest in time.
L. Rosasco and T. Poggio, “Convolutional layers build invariant and selective reproducing kernels,” CBMM Memo, in preparation
2015
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…