Fetching the paper…

Reading the bibliography…

2019

Sample-Efficient Neural Architecture Search by Learning Action Space

Wang, Linnan, Xie, Saining, Li, Teng et al.

Original
PDF
Post on X

Understand

Neural Architecture Search (NAS) has emerged as a promising technique for automatic neural network design.

However, existing MCTS based NAS approaches often utilize manually designed action space, which is not directly related to the performance metric to be optimized (e.g., accuracy), leading to sample-inefficient explorations of architectures.
To improve the sample efficiency, this paper proposes Latent Action Neural Architecture Search (LaNAS), which learns actions to recursively partition the search space into good or bad regions that contain networks with similar performance metrics.
During the search phase, as different action sequences lead to regions with different performance, the search efficiency can be significantly improved by biasing towards the good regions.

Sample-Efficient Neural Architecture Search by Learning Action Space · Around

Built on

N. Nayman, A. Noy, T. Ridnik, I. Friedman, R. Jin, and L. Zelnik, “Xnas: Neural architecture search with expert advice,” in Advances in Neural Information Processing Systems , 2019, pp. 1977–1987
1987
Earlier work this paper cites.
C. Mallows, “Letters to the editor,” The American Statistician , vol. 45, no. 3, pp. 256–262, 1991
1991
Earlier work this paper cites.
W. R. Gilks, N. G. Best, and K. Tan, “Adaptive rejection metropolis sampling within gibbs sampling,” Journal of the Royal Statistical Society: Series C (Applied Statistics) , vol. 44, no. 4, pp. 455–472, 1995
1995
Earlier work this paper cites.
W. Hörmann, “A rejection technique for sampling from t-concave distributions,” ACM Transactions on Mathematical Software (TOMS) , vol. 21, no. 2, pp. 182–193, 1995
1995
Earlier work this paper cites.
D. R. Jones, “A taxonomy of global optimization methods based on response surfaces,” Journal of global optimization , vol. 21, no. 4, pp. 345–383, 2001
2001
Earlier work this paper cites.
P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of the multiarmed bandit problem,” Machine learning , vol. 47, no. 2-3, pp. 235–256, 2002
2002
Earlier work this paper cites.
F. Hutter, “Automated configuration of algorithms for solving hard computational problems,” Ph.D. dissertation, University of British Columbia, 2009
2009
Earlier work this paper cites.
J. Villemonteix, E. Vazquez, and E. Walter, “An informational approach to the global optimization of expensive-to-evaluate functions,” Journal of Global Optimization , vol. 44, no. 4, p. 509, 2009
2009
Earlier work this paper cites.
J. S. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” in Advances in neural information processing systems , 2011, pp. 2546–2554
2011
Earlier work this paper cites.
D. Görür and Y. W. Teh, “Concave-convex adaptive rejection sampling,” Journal of Computational and Graphical Statistics , vol. 20, no. 3, pp. 670–691, 2011
2011
Earlier work this paper cites.
F. Hutter, H. H. Hoos, and K. Leyton-Brown, “Sequential model-based optimization for general algorithm configuration,” in International conference on learning and intelligent optimization . Springer, 2011, pp. 507–523
2011
Earlier work this paper cites.
C. Mansley, A. Weinstein, and M. Littman, “Sample-based planning for continuous action markov decision processes,” in Twenty-First International Conference on Automated Planning and Scheduling , 2011
2011
Earlier work this paper cites.
J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” in Advances in neural information processing systems , 2012, pp. 2951–2959
2012
Earlier work this paper cites.
A. Weinstein and M. L. Littman, “Bandit-based planning and learning in continuous-action markov decision processes,” in Twenty-Second International Conference on Automated Planning and Scheduling , 2012
2012
Earlier work this paper cites.
F. Hutter, H. Hoos, and K. Leyton-Brown, “An evaluation of sequential model-based optimization for expensive blackbox functions,” in Proceedings of the 15th annual conference companion on Genetic and evolutionary computation . ACM, 2013, pp. 1209–1216
2013
Earlier work this paper cites.
Z. Wang, M. Zoghi, F. Hutter, D. Matheson, and N. De Freitas, “Bayesian optimization in high dimensions via random embeddings,” in Twenty-Third International Joint Conference on Artificial Intelligence , 2013
2013
Earlier work this paper cites.
L. Buşoniu, A. Daniels, R. Munos, and R. Babuška, “Optimistic planning for continuous-action deterministic systems,” in 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) . IEEE, 2013, pp. 69–76
2013
Earlier work this paper cites.
Z. Wang, B. Shakibi, L. Jin, and N. de Freitas, “Bayesian multi- scale optimistic optimization,” 2014
2014
Earlier work this paper cites.
J. R. Gardner, M. J. Kusner, Z. E. Xu, K. Q. Weinberger, and J. P. Cunningham, “Bayesian optimization with inequality constraints.” in ICML , 2014, pp. 937–945
2014
Earlier work this paper cites.
R. Munos, “From bandits to monte-carlo tree search: The optimistic principle applied to optimization and planning,” technical report , vol. x, no. x, p. x, 2014
2014
Earlier work this paper cites.

Similar

B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” arXiv preprint arXiv:1611.01578 , 2016
Original
2016
Cited alongside, same era.
B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architectures using reinforcement learning,” arXiv preprint arXiv:1611.02167 , 2016
Original
2016
Cited alongside, same era.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” arXiv preprint arXiv:1603.06560 , 2016
Original
2016
Cited alongside, same era.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al. , “Mastering the game of go with deep neural networks and tree search,” nature , vol. 529, no. 7587, p. 484, 2016
2016
Cited alongside, same era.

Then

Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, and J. Sun, “Single path one-shot neural architecture search with uniform sampling,” arXiv preprint arXiv:1904.00420 , 2019
Original
2019
Closest in time.
M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” arXiv preprint arXiv:1905.11946 , 2019
Original
2019
Closest in time.
H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, “Once-for-all: Train one network and specialize it for efficient deployment,” arXiv preprint arXiv:1908.09791 , 2019
Original
2019
Closest in time.
C. Ying, A. Klein, E. Real, E. Christiansen, K. Murphy, and F. Hutter, “Nas-bench-101: Towards reproducible neural architecture search,” arXiv preprint arXiv:1902.09635 , 2019
Original
2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

S. Falkner, A. Klein, and F. Hutter, “Bohb: Robust and efficient hyperparameter optimization at scale,” arXiv preprint arXiv:1807.01774 , 2018

2018

Cited alongside, same era.

E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” arXiv preprint arXiv:1802.01548 , 2018

2018

Cited alongside, same era.

L. Wang, Y. Zhao, Y. Jinnai, and R. Fonseca, “Alphax: exploring neural architectures with deep neural networks and monte carlo tree search,” arXiv preprint arXiv:1805.07440 , 2018

Original

2018

Cited alongside, same era.

H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean, “Efficient neural architecture search via parameter sharing,” in International Conference on Machine Learning , 2018, pp. 4092–4101

2018

Cited alongside, same era.

H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,” arXiv preprint arXiv:1806.09055 , 2018

2018

Cited alongside, same era.

B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 8697–8710

2018

Cited alongside, same era.

C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 19–34

2018

Cited alongside, same era.

R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” in Advances in neural information processing systems , 2018, pp. 7816–7827

2018

Cited alongside, same era.

H. Zhou, M. Yang, J. Wang, and W. Pan, “Bayesnas: A bayesian approach for neural architecture search,” arXiv preprint arXiv:1905.04919 , 2019

2019

Closest in time.

X. Chen, L. Xie, J. Wu, and Q. Tian, “Progressive differentiable architecture search: Bridging the depth gap between search and evaluation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 1294–1303

2019

Closest in time.

Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong, “Pc-darts: Partial channel connections for memory-efficient architecture search,” arXiv preprint arXiv:1907.05737 , 2019

2019

Closest in time.

Y. Akimoto, S. Shirakawa, N. Yoshinari, K. Uchida, S. Saito, and K. Nishida, “Adaptive stochastic natural gradient method for one-shot neural architecture search,” arXiv preprint arXiv:1905.08537 , 2019

2019

Closest in time.

S. Xie, A. Kirillov, R. Girshick, and K. He, “Exploring randomly wired neural networks for image recognition,” arXiv preprint arXiv:1904.01569 , 2019

2019

Closest in time.

A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan et al. , “Searching for mobilenetv3,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 1314–1324

2019

Closest in time.

H. Shi, R. Pi, H. Xu, Z. Li, J. T. Kwok, and T. Zhang, “Multi-objective neural architecture search via predictive network performance optimization,” arXiv preprint arXiv:1911.09336 , 2019

2019

Closest in time.

Y. Guo, Y. Chen, Y. Zheng, P. Zhao, J. Chen, J. Huang, and M. Tan, “Breaking the curse of space explosion: Towards efficient nas with curriculum search,” in International Conference on Machine Learning . PMLR, 2020, pp. 3822–3831

2020

Closest in time.

X. Chu, T. Zhou, B. Zhang, and J. Li, “Fair darts: Eliminating unfair advantages in differentiable architecture search,” in European Conference on Computer Vision . Springer, 2020, pp. 465–480

2020

Closest in time.

A. Wan, X. Dai, P. Zhang, Z. He, Y. Tian, S. Xie, B. Wu, M. Yu, T. Xu, K. Chen et al. , “Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 12 965–12 974

2020

Closest in time.

J. Yu, P. Jin, H. Liu, G. Bender, P.-J. Kindermans, M. Tan, T. Huang, X. Song, R. Pang, and Q. Le, “Bignas: Scaling up neural architecture search with big single-stage models,” in European Conference on Computer Vision . Springer, 2020, pp. 702–717

2020

Closest in time.

X. Dai, A. Wan, P. Zhang, B. Wu, Z. He, Z. Wei, K. Chen, Y. Tian, M. Yu, P. Vajda et al. , “Fbnetv3: Joint architecture-recipe search using neural acquisition function,” arXiv preprint arXiv:2006.02049 , 2020

2020

Closest in time.

Y. Zhao, L. Wang, Y. Tian, R. Fonseca, and T. Guo, “Few-shot neural architecture search,” arXiv preprint arXiv:2006.06863 , 2020

2020

Closest in time.

L. Wang, R. Fonseca, and Y. Tian, “Learning search space partition for black-box optimization using monte carlo tree search,” Advances in Neural Information Processing Systems , 2020

2020

Closest in time.

M. Sazanovich, A. Nikolskaya, Y. Belousov, and A. Shpilman, “Solving black-box optimization challenge via learning search space partition for local bayesian optimization,” arXiv preprint arXiv:2012.10335 , 2020

2020

Closest in time.