2020

ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

Yang, Yibo, Li, Hongyang, You, Shan et al.

Understand

Neural architecture search (NAS) aims to produce the optimal sparse solution from a high-dimensional space spanned by all candidate connections.

Current gradient-based NAS methods commonly ignore the constraint of sparsity in the search phase, but project the optimized solution onto a sparse one by post-processing.
As a result, the dense super-net for search is inefficient to train and has a gap with the projected architecture for evaluation.
In this paper, we formulate neural architecture search as a sparse coding problem.

Built on

A new measure of rank correlation
M. G. Kendall · 1938
Earlier work this paper cites.
Regression shrinkage and selection via the lasso
R. Tibshirani · 1996
Earlier work this paper cites.
An iterative thresholding algorithm for linear inverse problems with a sparsity constraint
I. Daubechies, M. Defrise, and C. De Mol · 2004
Earlier work this paper cites.
Decoding by linear programming
E. J. Candes and T. Tao · 2005
Earlier work this paper cites.
A fast iterative shrinkage-thresholding algorithm for linear inverse problems
A. Beck and M. Teboulle · 2009
Earlier work this paper cites.
Introductory lectures on convex optimization: A basic course
Y. Nesterov · 2013
Earlier work this paper cites.
Cvx: Matlab software for disciplined convex programming, version 2.1, 2014
M. Grant and S. Boyd · 2014
Earlier work this paper cites.
Proximal algorithms
N. Parikh and S. Boyd · 2014
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
K. Simonyan and A. Zisserman · 2015
Earlier work this paper cites.
Going deeper with convolutions
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich · 2015
Earlier work this paper cites.
A survey of sparse representation: algorithms and applications
Z. Zhang, Y. Xu, J. Yang, X. Li, and D. Zhang · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Categorical reparameterization with gumbel-softmax
Original
E. Jang, S. Gu, and B. Poole · 2016
Earlier work this paper cites.
The concrete distribution: A continuous relaxation of discrete random variables
Original
C. J. Maddison, A. Mnih, and Y. W. Teh · 2016
Earlier work this paper cites.
Designing neural network architectures using reinforcemen learning
B. Baker, O. Gupta, N. Naik, and R. Raskar · 2017
Earlier work this paper cites.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam · 2017
Earlier work this paper cites.
Densely connected convolutional networks
G. Huang, Z. Liu, L. van der Maaten, and K. Weinberger · 2017
Earlier work this paper cites.
Large-scale evolution of image classifiers
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin · 2017
Earlier work this paper cites.
Learning from multiple teacher networks
S. You, C. Xu, C. Xu, and D. Tao · 2017
Earlier work this paper cites.
Neural architecture search with reinforcement learning
B. Zoph and Q. Le · 2017
Earlier work this paper cites.

Similar

Path-level network transformation for efficient architecture search
H. Cai, J. Yang, W. Zhang, S. Han, and Y. Yu · 2018
Cited alongside, same era.
Searching for efficient multi-scale architectures for dense image prediction
L.-C. Chen, M. Collins, Y. Zhu, G. Papandreou, B. Zoph, F. Schroff, H. Adam, and J. Shlens · 2018
Cited alongside, same era.
Neural architecture search with bayesian optimisation and optimal transport
K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing · 2018
Cited alongside, same era.
Optimization algorithm inspired deep neural network structure design
Original
H. Li, Y. Yang, D. Chen, and Z. Lin · 2018
Cited alongside, same era.
Hierarchical representations for efficient architecture search
H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu · 2018
Cited alongside, same era.
Neural architecture optimization
R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu · 2018
Cited alongside, same era.
Shufflenet v2: Practical guidelines for efficient cnn architecture design
N. Ma, X. Zhang, H.-T. Zheng, and J. Sun · 2018
Cited alongside, same era.
Efficient neural architecture search via parameter sharing
H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean · 2018
Cited alongside, same era.
Convolutional neural networks with alternately updated clique
Y. Yang, Z. Zhong, T. Shen, and Z. Lin · 2018
Cited alongside, same era.
You only search once: Single shot neural architecture search via direct sparse optimization
Original
X. Zhang, Z. Huang, and N. Wang · 2018
Cited alongside, same era.
Practical block-wise neural network architecture generation
Z. Zhong, J. Yan, W. Wu, J. Shao, and C.-L. Liu · 2018
Cited alongside, same era.
Learning transferable architectures for scalable image recognition
Original
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le · 2018
Cited alongside, same era.

Then

Single path one-shot neural architecture search with uniform sampling
Original
Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun · 2019
Later among the works it cites.
Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation
C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille, and L. Fei-Fei · 2019
Later among the works it cites.
Darts: Differentiable architecture search
H. Liu, K. Simonyan, and Y. Yang · 2019
Later among the works it cites.
Evolving deep neural networks
R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, H. Shahrzad, A. Navruzyan, N. Duffy, et al · 2019
Later among the works it cites.
Efficient neural architecture transformation search in channel-level for object detection
J. Peng, M. Sun, Z.-X. ZHANG, T. Tan, and J. Yan · 2019
Later among the works it cites.
Regularized evolution for image classifier architecture search
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le · 2019
Later among the works it cites.
Mnasnet: Platform-aware neural architecture search for mobile
M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le · 2019
Later among the works it cites.
Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search
B. Wu, X. Dai, P. Zhang, Y. Wang, F. Sun, Y. Wu, Y. Tian, P. Vajda, Y. Jia, and K. Keutzer · 2019
Later among the works it cites.
Snas: stochastic neural architecture search
S. Xie, H. Zheng, C. Liu, and L. Lin · 2019
Later among the works it cites.
BayesNAS: A Bayesian approach for neural architecture search
H. Zhou, M. Yang, J. Wang, and W. Pan · 2019
Later among the works it cites.
Once-for-all: Train one network and specialize it for efficient deployment
H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han · 2020
Closest in time.
Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning
Y. Gao, H. Bai, Z. Jie, J. Ma, K. Jia, and W. Liu · 2020
Closest in time.
Dsnas: Direct neural architecture search without parameter retraining
S. Hu, S. Xie, H. Zheng, C. Liu, J. Shi, X. Liu, and D. Lin · 2020
Closest in time.
Atomnas: Fine-grained end-to-end neural architecture search
J. Mei, Y. Li, X. Lian, X. Jin, L. Yang, A. Yuille, and J. Yang · 2020
Closest in time.
Efficientdet: Scalable and efficient object detection
M. Tan, R. Pang, and Q. V. Le · 2020
Closest in time.
Pc-darts: Partial channel connections for memory-efficient differentiable architecture search
Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong · 2020
Closest in time.
Nas evaluation is frustratingly hard
A. Yang, P. M. Esperança, and F. M. Carlucci · 2020
Closest in time.
Dynamical system inspired adaptive time stepping controller for residual network families
Y. Yang, J. Wu, H. Li, X. Li, T. Shen, and Z. Lin · 2020
Closest in time.
Efficient neural architecture search via proximal iterations
Q. Yao, J. Xu, W.-W. Tu, and Z. Zhu · 2020
Closest in time.
Greedynas: Towards fast one-shot nas with greedy supernet
S. You, T. Huang, M. Yang, F. Wang, C. Qian, and C. Zhang · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…