2020

Neural Kernels Without Tangents

Shankar, Vaishaal, Fang, Alex, Guo, Wenshuo et al.

Understand

We investigate the connections between neural networks and simple building blocks in kernel space.

In particular, using well established feature space tools such as direct sum, averaging, and moment lifting, we present an algebra for creating "compositional" kernels from bags of features.
We show that these operations correspond to many of the building blocks of "neural tangent kernels (NTK)".
Experimentally, we show that there is a correlation in test error between neural network architectures and the associated kernels.

Built on

Benchmarking optimization software with performance profiles
Dolan, E. D. and Moré, J. J · 2002
Earlier work this paper cites.
Kernel Methods for Pattern Analysis
Shawe-Taylor, J., Cristianini, N., et al · 2004
Earlier work this paper cites.
Statistical comparisons of classifiers over multiple data sets
Demšar, J · 2006
Earlier work this paper cites.
Random features for large-scale kernel machines
Rahimi, A. and Recht, B · 2008
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2009
Earlier work this paper cites.
Learning feature representations with k-means
Coates, A. and Ng, A. Y · 2012
Earlier work this paper cites.
Maxout networks
Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A. C., and Bengio, Y · 2013
Earlier work this paper cites.
Scalable kernel methods via doubly stochastic gradients
Dai, B., Xie, B., He, N., Liang, Y., Raj, A., Balcan, M., and Song, L · 2014
Earlier work this paper cites.
Do we need hundreds of classifiers to solve real world classification problems?
Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D · 2014
Earlier work this paper cites.
Convolutional kernel networks
Mairal, J., Koniusz, P., Harchaoui, Z., and Schmid, C · 2014
Earlier work this paper cites.

Similar

Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity
Daniely, A., Frostig, R., and Singer, Y · 2016
Cited alongside, same era.
Learning to compose domain-specific transformations for data augmentation
Ratner, A., Ehrenberg, H., Hussain, Z., Dunnmon, J., and Re, C · 2017
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Jacot, A., Hongler, C., and Gabriel, F · 2018
Cited alongside, same era.
Deep neural networks as gaussian processes
Lee, J., Bahri, Y., Novak, R., Schoenholz, S. S., Pennington, J., and Sohl-Dickstein, J · 2018
Cited alongside, same era.
Kernel machines that adapt to gpus for effective large batch training, 2018
Ma, S. and Belkin, M · 2018
Cited alongside, same era.
myrtle.ai, 2018
Page, D · 2018
Cited alongside, same era.
numpywren: Serverless linear algebra
Original
Shankar, V., Krauth, K., Pu, Q., Jonas, E., Venkataraman, S., Stoica, I., Recht, B., and Ragan-Kelley, J · 2018
Cited alongside, same era.
Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions
Original
Vasilache, N., Zinenko, O., Theodoridis, T., Goyal, P., DeVito, Z., Moses, W. S., Verdoolaege, S., Adams, A., and Cohen, A · 2018
Cited alongside, same era.
Learning and generalization in overparameterized neural networks, going beyond two layers
Allen-Zhu, Z., Li, Y., and Liang, Y · 2019
Cited alongside, same era.
On exact computation with an infinitely wide neural net
Arora, S., Du, S. S., Hu, W., Li, Z., Salakhutdinov, R., and Wang, R · 2019
Cited alongside, same era.
Gradient descent finds global minima of deep neural networks
Du, S. S., Lee, J. D., Li, H., Wang, L., and Zhai, X
Cited in the paper.
Gradient descent provably optimizes over-parameterized neural networks
Du, S. S., Zhai, X., Poczos, B., and Singh, A
Cited in the paper.

Then

Randaugment: Practical data augmentation with no separate search
Original
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V · 2019
Later among the works it cites.
A kernel theory of modern data augmentation
Dao, T., Gu, A., Ratner, A. J., Smith, V., Sa, C. D., and Re, C · 2019
Later among the works it cites.
Linearized two-layers neural networks in high dimension, 2019
Ghorbani, B., Mei, S., Misiakiewicz, T., and Montanari, A · 2019
Later among the works it cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J · 2019
Later among the works it cites.
Enhanced convolutional neural tangent kernels, 2019
Li, Z., Wang, R., Yu, D., Du, S. S., Hu, W., Salakhutdinov, R., and Arora, S · 2019
Later among the works it cites.
Bayesian deep convolutional networks with many channels are gaussian processes
Novak, R., Xiao, L., Bahri, Y., Lee, J., Yang, G., Hron, J., Abolafia, D. A., Pennington, J., and Sohl-Dickstein, J · 2019
Later among the works it cites.
Do imagenet classifiers generalize to imagenet?
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V · 2019
Later among the works it cites.
Exact gaussian processes on a million data points
Wang, K. A., Pleiss, G., Gardner, J., Tyree, S., Weinberger, K., and Wilson, A. G · 2019
Later among the works it cites.
Harnessing the power of infinitely wide deep nets on small-data tasks
Arora, S., Du, S. S., Li, Z., Salakhutdinov, R., Wang, R., and Yu, D · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…