Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks

Arora, S., Du, S. S., Hu, W., Li, Z. & Wang, R. (2019), ‘Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks’, arXiv preprint arXiv:1901.08584

Original

1901

Earlier work this paper cites.

Xu, Z.-Q. J., Zhang, Y., Luo, T., Xiao, Y. & Ma, Z. (2019), ‘Frequency principle: Fourier analysis sheds light on deep neural networks’, arXiv preprint arXiv:1901.06523

Original

1901

Earlier work this paper cites.

Cao, Y. & Gu, Q. (2019), ‘A generalization theory of gradient descent for learning over-parameterized deep relu networks’, arXiv preprint arXiv:1902.01384

Original

1902

Earlier work this paper cites.

Lee, J., Xiao, L., Schoenholz, S. S., Bahri, Y., Sohl-Dickstein, J. & Pennington, J. (2019), ‘Wide neural networks of any depth evolve as linear models under gradient descent’, arXiv preprint arXiv:1902.06720

Original

1902

Earlier work this paper cites.

Mei, S., Misiakiewicz, T. & Montanari, A. (2019), ‘Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit’, arXiv preprint arXiv:1902.06015

Original

1902

Earlier work this paper cites.

E, W., Ma, C., Wang, Q. & Wu, L. (2019), ‘Analysis of the gradient descent algorithm for a deep neural network model with skip-connections’, arXiv preprint arXiv:1904.05263

Original

1904

Earlier work this paper cites.

E, W., Ma, C. & Wu, L. (2019), ‘A comparative analysis of the optimization and generalization property of two-layer neural network and random feature models under gradient descent dynamics’, arXiv preprint arXiv:1904.04326

Original

1904

Earlier work this paper cites.

arXiv: 1905.07777. http://arxiv.org/abs/1905.07777

Original

Zhang, Y., Xu, Z.-Q. J., Luo, T. & Ma, Z. (2019), ‘A type of generalization error induced by initialization in deep neural networks’, arXiv:1905.07777 [cs, stat] · 1905

Earlier work this paper cites.

Cybenko, G. (1989), ‘Approximation by superpositions of a sigmoidal function’, Mathematics of control, signals and systems

1989

Earlier work this paper cites.

Hochreiter, S. & Schmidhuber, J. (1995), Simplifying neural nets by discovering flat minima, in

1995

Earlier work this paper cites.

Bartlett, P. L., Maiorov, V. & Meir, R. (1999), Almost linear vc dimension bounds for piecewise polynomial networks, in

1999

Earlier work this paper cites.

Bartlett, P. L. & Mendelson, S. (2002), ‘Rademacher and gaussian complexities: Risk bounds and structural results’, Journal of Machine Learning Research

2002

Earlier work this paper cites.

Bousquet, O. & Elisseeff, A. (2002), ‘Stability and generalization’, Journal of machine learning research

2002

Earlier work this paper cites.

Xu, H. & Mannor, S. (2012), ‘Robustness and generalization’, Machine learning

2012

Earlier work this paper cites.

Kingma, D. P. & Ba, J. (2014), ‘Adam: A method for stochastic optimization’, arXiv preprint arXiv:1412.6980

Original

2014

Earlier work this paper cites.

Neyshabur, B., Tomioka, R. & Srebro, N. (2014), ‘In search of the real inductive bias: On the role of implicit regularization in deep learning’, arXiv preprint arXiv:1412.6614

Original

2014

Cited alongside, same era.

Shalev-Shwartz, S. & Ben-David, S. (2014), Understanding machine learning: From theory to algorithms

2014

Cited alongside, same era.

Hardt, M., Recht, B. & Singer, Y. (2015), ‘Train faster, generalize better: Stability of stochastic gradient descent’, arXiv preprint arXiv:1509.01240

Original

2015

Cited alongside, same era.

LeCun, Y., Bengio, Y. & Hinton, G. (2015), ‘Deep learning’, nature

2015

Cited alongside, same era.

Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M. & Tang, P. T. P. (2016), ‘On large-batch training for deep learning: Generalization gap and sharp minima’, arXiv preprint arXiv:1609.04836

Original

Wu, L., Zhu, Z. & E, W. (2017), ‘Towards understanding generalization of deep learning: Perspective of loss landscapes’, arXiv preprint arXiv:1706.10239

Original

2017

Later among the works it cites.

E, W., Ma, C. & Wu, L. (2018), ‘A priori estimates of the generalization error for two-layer neural networks’, arXiv preprint arXiv:1810.06397

Original

2018

Later among the works it cites.

Jacot, A., Gabriel, F. & Hongler, C. (2018), Neural tangent kernel: Convergence and generalization in neural networks, in

2018

Later among the works it cites.

Mei, S., Montanari, A. & Nguyen, P.-M. (2018), ‘A mean field view of the landscape of two-layer neural networks’, Proceedings of the National Academy of Sciences

2018

Later among the works it cites.

Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y. & Srebro, N. (2018), ‘Towards understanding the role of over-parametrization in generalization of neural networks’, arXiv preprint arXiv:1805.12076