2019

BayesNAS: A Bayesian Approach for Neural Architecture Search

Zhou, Hongpeng, Yang, Minghao, Wang, Jun et al.

Understand

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training.

It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network.
However, there are two issues associated with most one-shot NAS methods.
First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations.

Built on

A back-propagation algorithm with optimal use of hidden units
Chauvin, Y · 1989
Earlier work this paper cites.
Optimal brain damage
LeCun, Y., Denker, J. S., and Solla, S. A · 1990
Earlier work this paper cites.
Generalization by weight-elimination with application to forecasting
Weigend, A. S., Rumelhart, D. E., and Huberman, B. A · 1991
Earlier work this paper cites.
Optimal brain surgeon and general network pruning
Hassibi, B., Stork, D. G., and Wolff, G. J · 1993
Earlier work this paper cites.
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E. and Van Camp, D · 1993
Earlier work this paper cites.
Bayesian learning for neural networks
Neal, R. M · 1995
Earlier work this paper cites.
Structural learning with forgetting
Ishikawa, M · 1996
Earlier work this paper cites.
Bayesian methods for backpropagation networks
MacKay, D. J · 1996
Earlier work this paper cites.
Natural gradient works efficiently in learning
Amari, S.-I · 1998
Earlier work this paper cites.
The mnist database of handwritten digits
LeCun, Y · 1998
Earlier work this paper cites.
Sparse bayesian learning and the relevance vector machine
Tipping, M. E · 2001
Earlier work this paper cites.
Convex optimisation
Boyd, S. and Vandenberghe, L · 2004
Earlier work this paper cites.
Numerical Optimization
Nocedal, J. and Wright, S. J · 2006
Earlier work this paper cites.
Enhancing sparsity by reweighted ℓ 1 \ell_{1} minimization
Candes, E. J., Wakin, M. B., and Boyd, S. P · 2008
Earlier work this paper cites.
Practical variational inference for neural networks
Graves, A · 2011
Earlier work this paper cites.
Statistical decision theory and Bayesian analysis
Berger, J. O · 2013
Earlier work this paper cites.
Expectation propagation for neural networks with sparsity-promoting priors
Jylänki, P., Nummenmaa, A., and Vehtari, A · 2014
Earlier work this paper cites.
Binaryconnect: Training deep neural networks with binary weights during propagations
Courbariaux, M., Bengio, Y., and David, J.-P · 2015
Earlier work this paper cites.
Probabilistic backpropagation for scalable learning of bayesian neural networks
Hernández-Lobato, J. M. and Adams, R · 2015
Earlier work this paper cites.

Similar

Optimizing neural networks with kronecker-factored approximate curvature
Martens, J. and Grosse, R · 2015
Cited alongside, same era.
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A · 2015
Cited alongside, same era.
Dynamic network surgery for efficient dnns
Guo, Y., Yao, A., and Chen, Y · 2016
Cited alongside, same era.
Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding
Han, S., Mao, H., and Dally, W. J · 2016
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Convolutional neural fabrics
Saxena, S. and Verbeek, J · 2016
Cited alongside, same era.
Learning structured sparsity in deep neural networks
Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H · 2016
Cited alongside, same era.
Designing neural network architectures using reinforcement learning
Baker, B., Gupta, O., Naik, N., and Raskar, R · 2017
Cited alongside, same era.
Practical gauss-newton optimisation for deep learning
Botev, A., Ritter, H., and Barber, D · 2017
Cited alongside, same era.
Improved regularization of convolutional neural networks with cutout, 2017
DeVries, T. and Taylor, G. W · 2017
Cited alongside, same era.
Deep pyramidal residual networks
Han, D., Kim, J., and Kim, J · 2017
Cited alongside, same era.
Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H · 2017
Cited alongside, same era.

Then

Soft weight-sharing for neural network compression
Ullrich, K., Meeds, E., and Welling, M · 2017
Later among the works it cites.
Genetic cnn
Xie, L. and Yuille, A · 2017
Later among the works it cites.
Neural architecture search with reinforcement learning
Zoph, B. and Le, Q. V · 2017
Later among the works it cites.
Understanding and simplifying one-shot architecture search
Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., and Le, Q · 2018
Later among the works it cites.
SMASH: One-shot model architecture search through hypernetworks
Brock, A., Lim, T., Ritchie, J., and Weston, N · 2018
Later among the works it cites.
Path-level network transformation for efficient architecture search
Cai, H., Yang, J., Zhang, W., Han, S., and Yu, Y · 2018
Later among the works it cites.
Morphnet: Fast & simple resource-constrained structure learning of deep networks
Gordon, A., Eban, E., Nachum, O., Chen, B., Wu, H., Yang, T.-J., and Choi, E · 2018
Later among the works it cites.
Learning sparse neural networks through l 0 l_{0} regularization
Louizos, C., Welling, M., and Kingma, D. P · 2018
Later among the works it cites.
Efficient neural architecture search via parameter sharing
Pham, H. Q., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J · 2018
Later among the works it cites.
Shufflenet: An extremely efficient convolutional neural network for mobile devices
Zhang, X., Zhou, X., Lin, M., and Sun, J · 2018
Later among the works it cites.
Practical block-wise neural network architecture generation
Zhong, Z., Yan, J., Wu, W., Shao, J., and Liu, C.-L · 2018
Later among the works it cites.
Learning transferable architectures for scalable image recognition
Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V · 2018
Later among the works it cites.
ProxylessNAS: Direct neural architecture search on target task and hardware
Original
Cai, H., Zhu, L., and Han, S · 2019
Closest in time.
Bayesian learning of neural network architectures
Dikov, G., van der Smagt, P., and Bayer, J · 2019
Closest in time.
Snip: Single-shot network pruning based on connection sensitivity
Lee, N., Ajanthan, T., and Torr, P. H. S · 2019
Closest in time.
Evolving deep neural networks
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al · 2019
Closest in time.
Regularized evolution for image classifier architecture search
Real, E., Aggarwal, A., Huang, Y., and Le, Q. V · 2019
Closest in time.
SNAS: stochastic neural architecture search
Xie, S., Zheng, H., Liu, C., and Lin, L · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…