2020

Evolving Search Space for Neural Architecture Search

Ci, Yuanzheng, Lin, Chen, Sun, Ming et al.

Understand

The automation of neural architecture design has been a coveted alternative to human experts.

Recent works have small search space, which is easier to optimize but has a limited upper bound of the optimal solution.
Extra human design is needed for those methods to propose a more suitable space with respect to the specific task and algorithm capacity.
To further enhance the degree of automation for neural architecture search, we present a Neural Search-space Evolution (NSE) scheme that iteratively amplifies the results from the previous effort by maintaining an optimized search space subset.

Built on

The task rehearsal method of life-long learning: Overcoming impoverished data
Daniel L Silver and Robert E Mercer · 2002
Earlier work this paper cites.
Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures
James Bergstra, Daniel Yamins, and David Daniel Cox · 2013
Earlier work this paper cites.
Microsoft coco: Common objects in context
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick · 2014
Earlier work this paper cites.
Binaryconnect: Training deep neural networks with binary weights during propagations
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David · 2015
Earlier work this paper cites.
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Earlier work this paper cites.
Towards automatically-tuned neural networks
Hector Mendoza, Aaron Klein, Matthias Feurer, Jost Tobias Springenberg, and Frank Hutter · 2016
Earlier work this paper cites.
Neural architecture search with reinforcement learning
Original
Barret Zoph and Quoc V Le · 2016
Earlier work this paper cites.
Mobilenets: Efficient convolutional neural networks for mobile vision applications
Original
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam · 2017
Earlier work this paper cites.
Searching for activation functions
Original
Prajit Ramachandran, Barret Zoph, and Quoc V Le · 2017
Earlier work this paper cites.
Large-scale evolution of image classifiers
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc V Le, and Alexey Kurakin · 2017
Earlier work this paper cites.
Inception-v4, inception-resnet and the impact of residual connections on learning
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi · 2017
Earlier work this paper cites.
Aggregated residual transformations for deep neural networks
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He · 2017
Earlier work this paper cites.
Understanding and simplifying one-shot architecture search
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le · 2018
Earlier work this paper cites.
Proxylessnas: Direct neural architecture search on target task and hardware
Original
Han Cai, Ligeng Zhu, and Song Han · 2018
Earlier work this paper cites.
Squeeze-and-excitation networks
Jie Hu, Li Shen, and Gang Sun · 2018
Earlier work this paper cites.
Progressive neural architecture search
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy · 2018
Earlier work this paper cites.
Darts: Differentiable architecture search
Original
Hanxiao Liu, Karen Simonyan, and Yiming Yang · 2018
Earlier work this paper cites.
Nsga-net: A multi-objective genetic algorithm for neural architecture search
Original
Zhichao Lu, Ian Whalen, Vishnu Boddeti, Yashesh Dhebar, Kalyanmoy Deb, Erik Goodman, and Wolfgang Banzhaf · 2018
Earlier work this paper cites.
Shufflenet v2: Practical guidelines for efficient cnn architecture design
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun · 2018
Earlier work this paper cites.
Efficient neural architecture search via parameter sharing
Original
Hieu Pham, Melody Y Guan, Barret Zoph, Quoc V Le, and Jeff Dean · 2018
Earlier work this paper cites.
Regularized evolution for image classifier architecture search
Original
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le · 2018
Earlier work this paper cites.

Similar

Mobilenetv2: Inverted residuals and linear bottlenecks
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen · 2018
Cited alongside, same era.
Mnasnet: Platform-aware neural architecture search for mobile
Original
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V Le · 2018
Cited alongside, same era.
Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search
Original
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer · 2018
Cited alongside, same era.
Netadapt: Platform-aware neural network adaptation for mobile applications
Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam · 2018
Cited alongside, same era.
Slimmable neural networks
Original
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang · 2018
Cited alongside, same era.
You only search once: Single shot neural architecture search via direct sparse optimization
Original
Xinbang Zhang, Zehao Huang, and Naiyan Wang · 2018
Cited alongside, same era.
Practical block-wise neural network architecture generation
Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu · 2018
Cited alongside, same era.
Learning transferable architectures for scalable image recognition
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le · 2018
Cited alongside, same era.
Once-for-all: Train one network and specialize it for efficient deployment
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han · 2019
Cited alongside, same era.
Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search
Original
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li · 2019
Cited alongside, same era.
Betanas: Balanced training and selective drop for neural architecture search
Original
Muyuan Fang, Qiang Wang, and Zhao Zhong · 2019
Cited alongside, same era.
Searching for mobilenetv3
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al · 2019
Cited alongside, same era.

Then

Evaluating the search phase of neural architecture search
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann · 2019
Later among the works it cites.
Hardware-aware one-shot neural architecture search in coordinate ascent framework
Original
Li Lyna Zhang, Yuqing Yang, Yuhang Jiang, Wenwu Zhu, and Yunxin Liu · 2019
Later among the works it cites.
Learning data augmentation strategies for object detection
Original
Barret Zoph, Ekin D Cubuk, Golnaz Ghiasi, Tsung-Yi Lin, Jonathon Shlens, and Quoc V Le · 2019
Later among the works it cites.
Can weight sharing outperform random architecture search? an investigation with tunas
Gabriel Bender, Hanxiao Liu, Bo Chen, Grace Chu, Shuyang Cheng, Pieter-Jan Kindermans, and Quoc V Le · 2020
Closest in time.
What is the state of neural network pruning?
Original
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag · 2020
Closest in time.
Fbnetv3: Joint architecture-recipe search using neural acquisition function
Original
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al · 2020
Closest in time.
Single path one-shot neural architecture search with uniform sampling
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun · 2020
Closest in time.
Angle-based search space shrinking for neural architecture search
Original
Yiming Hu, Yuding Liang, Zichao Guo, Ruosi Wan, Xiangyu Zhang, Yichen Wei, Qingyi Gu, and Jian Sun · 2020
Closest in time.
Improving one-shot nas by suppressing the posterior fading
Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang · 2020
Closest in time.
Efficientdet: Scalable and efficient object detection
Mingxing Tan, Ruoming Pang, and Quoc V Le · 2020
Closest in time.
Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al · 2020
Closest in time.
Fp-nas: Fast probabilistic neural architecture search
Original
Zhicheng Yan, Xiaoliang Dai, Peizhao Zhang, Yuandong Tian, Bichen Wu, and Matt Feiszli · 2020
Closest in time.
Greedynas: Towards fast one-shot nas with greedy supernet
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang · 2020
Closest in time.
Bignas: Scaling up neural architecture search with big single-stage models
Original
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le · 2020
Closest in time.
How does supernet help in neural architecture search?
Original
Yuge Zhang, Quanlu Zhang, and Yaming Yang · 2020
Closest in time.
Econas: Finding proxies for economical neural architecture search
Dongzhan Zhou, Xinchi Zhou, Wenwei Zhang, Chen Change Loy, Shuai Yi, Xuesen Zhang, and Wanli Ouyang · 2020
Closest in time.
Bn-nas: Neural architecture search with batch normalization
Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, and Wanli Ouyang · 2021
Closest in time.
Glit: Neural architecture search for global and local image transformer
Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Wanli Ouyang, et al · 2021
Closest in time.
Inception convolution with efficient dilation search
Jie Liu, Chuming Li, Feng Liang, Chen Lin, Ming Sun, Junjie Yan, Wanli Ouyang, and Dong Xu · 2021
Closest in time.
Towards high performance extremely low-bit neural networks
Mingzhu Shen, Feng Liang, Ruihao Gong, Yuhang Li, Chuming Li, Chen Lin, Fengwei Yu, Junjie Yan, and Wanli Ouyang · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…