Fetching the paper…

Reading the bibliography…

2019

AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

Yu, Jiahui, Huang, Thomas

Original
PDF
Post on X

Understand

We study how to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size).

A simple and one-shot solution, named AutoSlim, is presented.
Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations.
We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop.

AutoSlim: Towards One-Shot Architecture Search for Channel Numbers · Around

Built on

Z. Shen, Z. Liu, J. Li, Y.-G. Jiang, Y. Chen, and X. Xue, “Dsod: Learning deeply supervised object detectors from scratch,” in
1927
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner
1998
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in
2009
Earlier work this paper cites.
A. Krizhevsky, “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009
2009
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in
2012
Earlier work this paper cites.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”
Original
2014
Earlier work this paper cites.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,”
Original
2015
Earlier work this paper cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in
2015
Earlier work this paper cites.
S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in
2015
Earlier work this paper cites.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”
Original
2015
Earlier work this paper cites.
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,”
Original
2015
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in
2016
Earlier work this paper cites.
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in
2016
Earlier work this paper cites.
J. Yu, Y. Jiang, Z. Wang, Z. Cao, and T. Huang, “Unitbox: An advanced object detection network,” in
2016
Earlier work this paper cites.
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,”
Original
2016
Earlier work this paper cites.
B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,”
Original
2016
Earlier work this paper cites.
Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang, “Learning efficient convolutional networks through network slimming,” in
2017
Earlier work this paper cites.
G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Q. Weinberger, “Multi-scale dense networks for resource efficient image classification,”
Original
2017
Earlier work this paper cites.
X. Wang, F. Yu, Z.-Y. Dou, and J. E. Gonzalez, “Skipnet: Learning dynamic routing in convolutional networks,”
Original
2017
Earlier work this paper cites.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,”
Original
2017
Earlier work this paper cites.

Similar

X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,”
Original
2017
Cited alongside, same era.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in
2017
Cited alongside, same era.
C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in
2017
Cited alongside, same era.
D. Han, J. Kim, and J. Kim, “Deep pyramidal residual networks,” in
2017
Cited alongside, same era.
K. Zhang, L. Guo, C. Gao, and Z. Zhao, “Pyramidal ror for image classification,”
2017
Cited alongside, same era.

Then

H. Cai, L. Zhu, and S. Han, “Proxylessnas: Direct neural architecture search on target task and hardware,”
Original
2018
Later among the works it cites.
Z. Liu, M. Sun, T. Zhou, G. Huang, and T. Darrell, “Rethinking the value of network pruning,”
Original
2018
Later among the works it cites.
Z. Huang and N. Wang, “Data-driven sparse structure selection for deep neural networks,” in
2018
Later among the works it cites.
J. Yu, L. Yang, N. Xu, J. Yang, and T. Huang, “Slimmable neural networks,”
Original
2018
Later among the works it cites.
Y. He, J. Lin, Z. Liu, H. Wang, L.-J. Li, and S. Han, “Amc: Automl for model compression and acceleration on mobile devices,” in
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

J.-H. Luo, J. Wu, and W. Lin, “Thinet: A filter level pruning method for deep neural network compression,”

2017

Cited alongside, same era.

Y. He, X. Zhang, and J. Sun, “Channel pruning for accelerating very deep neural networks,” in

2017

Cited alongside, same era.

J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”

2017

Cited alongside, same era.

N. Heess, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. Eslami, M. Riedmiller

2017

Cited alongside, same era.

H. Liu, K. Simonyan, O. Vinyals, C. Fernando, and K. Kavukcuoglu, “Hierarchical representations for efficient architecture search,”

2017

Cited alongside, same era.

A. Brock, T. Lim, J. M. Ritchie, and N. Weston, “Smash: one-shot model architecture search through hypernetworks,”

2017

Cited alongside, same era.

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in

2017

Cited alongside, same era.

T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. Sandler, V. Sze, and H. Adam, “Netadapt: Platform-aware neural network adaptation for mobile applications,” in

2018

Later among the works it cites.

J. Ye, X. Lu, Z. Lin, and J. Z. Wang, “Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers,”

2018

Later among the works it cites.

Q. Huang, K. Zhou, S. You, and U. Neumann, “Learning to prune filters in convolutional neural networks,” in

2018

Later among the works it cites.

N. Lee, T. Ajanthan, and P. H. Torr, “Snip: Single-shot network pruning based on connection sensitivity,”

2018

Later among the works it cites.

J. Frankle and M. Carbin, “The lottery ticket hypothesis: Training pruned neural networks,”

2018

Later among the works it cites.

T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,”

2018

Later among the works it cites.

G. Bender, P.-J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le, “Understanding and simplifying one-shot architecture search,” in

2018

Later among the works it cites.

H. Pham, M. Y. Guan, B. Zoph, Q. V. Le, and J. Dean, “Efficient neural architecture search via parameter sharing,”

2018

Later among the works it cites.

B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in

2018

Later among the works it cites.

C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in

2018

Later among the works it cites.

H. Liu, K. Simonyan, and Y. Yang, “Darts: Differentiable architecture search,”

2018

Later among the works it cites.

R. Luo, F. Tian, T. Qin, E. Chen, and T.-Y. Liu, “Neural architecture optimization,” in

2018

Later among the works it cites.

C. Zhang, M. Ren, and R. Urtasun, “Graph hypernetworks for neural architecture search,”

2018

Later among the works it cites.

J. Yu and T. Huang, “Universally slimmable networks and improved training techniques,”

2019

Closest in time.