2017

NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Dai, Xiaoliang, Yin, Hongxu, Jha, Niraj K.

Understand

Deep neural networks (DNNs) have begun to have a pervasive impact on various applications of machine learning.

However, the problem of finding an optimal DNN architecture for large applications is challenging.
Common approaches go for deeper and larger DNN architectures but may incur substantial redundancy.
To address these problems, we introduce a network growth algorithm that complements network pruning to learn both weights and compact DNN architectures during training.

Built on

T. Ash, “Dynamic node creation in backpropagation networks,” Connection Science , vol. 1, no. 4, pp. 365–375, 1989
1989
Earlier work this paper cites.
M. Mezard and J.-P. Nadal, “Learning in feedforward layered networks: The tiling algorithm,” J. of Physics A: Mathematical and General , vol. 22, no. 12, p. 2191, 1989
1989
Earlier work this paper cites.
S. Lowel and W. Singer, “Selection of intrinsic horizontal connections in the visual cortex by correlated neuronal activity,” Science , vol. 255, no. 5041, p. 209, 1992
1992
Earlier work this paper cites.
C. J. Burges and B. Schölkopf, “Improving the accuracy and speed of support vector machines,” in Proc. Advances in Neural Information Processing Systems , 1997, pp. 375–381
1997
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998
1998
Earlier work this paper cites.
K. O. Stanley and R. Miikkulainen, “Evolving neural networks through augmenting topologies,” Evolutionary Computation , vol. 10, no. 2, pp. 99–127, 2002
2002
Earlier work this paper cites.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “ImageNet: A large-scale hierarchical image database,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2009, pp. 248–255
2009
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in Proc. Advances in Neural Information Processing Systems , vol. 2011, no. 2, 2011, p. 5
2011
Earlier work this paper cites.
D. Yu, F. Seide, G. Li, and L. Deng, “Exploiting sparseness in deep neural networks for large vocabulary speech recognition,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing , 2012, pp. 4409–4412
2012
Earlier work this paper cites.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” in Proc. Advances in Neural Information Processing Systems , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
A. Graves, A.-R. Mohamed, and G. E. Hinton, “Speech recognition with deep recurrent neural networks,” in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing , 2013, pp. 6645–6649
2013
Earlier work this paper cites.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Proc. Advances in Neural Information Processing Systems , 2014, pp. 3104–3112
2014
Earlier work this paper cites.
Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115 , 2014
Original
2014
Earlier work this paper cites.
A. Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” arXiv preprint arXiv:1404.5997 , 2014
Original
2014
Earlier work this paper cites.
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast feature embedding,” in Proc. ACM Int. Conf. Multimedia , 2014, pp. 675–678
2014
Earlier work this paper cites.
M. D. Collins and P. Kohli, “Memory bounded deep convolutional networks,” arXiv preprint arXiv:1412.1442 , 2014
Original
2014
Earlier work this paper cites.

Similar

E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus, “Exploiting linear structure within convolutional networks for efficient evaluation,” in Proc. Advances in Neural Information Processing Systems , 2014, pp. 1269–1277
2014
Cited alongside, same era.
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Cited alongside, same era.
Y. LeCun, Y. Bengio, and G. E. Hinton, “Deep learning,” Nature , vol. 521, no. 7553, pp. 436–444, 2015
2015
Cited alongside, same era.
S. Han, J. Pool, J. Tran, and W. J. Dally, “Learning both weights and connections for efficient neural network,” in Proc. Advances in Neural Information Processing Systems , 2015, pp. 1135–1143
2015
Cited alongside, same era.
B. Liu, M. Wang, H. Foroosh, M. Tappen, and M. Pensky, “Sparse convolutional neural networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2015, pp. 806–814
2015
Cited alongside, same era.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. Int. Conf. Machine Learning , 2015, pp. 448–456
2015
Cited alongside, same era.
S. Srinivas and R. V. Babu, “Data-free parameter pruning for deep neural networks,” arXiv preprint arXiv:1507.06149 , 2015
Original
2015
Cited alongside, same era.
Z. Yang, M. Moczulski, M. Denil, N. de Freitas, A. Smola, L. Song, and Z. Wang, “Deep fried ConvNets,” in Proc. IEEE Int. Conf. Computer Vision , 2015, pp. 1476–1483
2015
Cited alongside, same era.
G. Leisman, R. Mualem, and S. K. Mughrabi, “The neurological development of the child with the educational enrichment in mind,” Psicología Educativa , vol. 21, no. 2, pp. 79–96, 2015
2015
Cited alongside, same era.
B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” arXiv preprint arXiv:1611.01578 , 2016
Original
2016
Cited alongside, same era.
B. Baker, O. Gupta, N. Naik, and R. Raskar, “Designing neural network architectures using reinforcement learning,” arXiv preprint arXiv:1611.02167 , 2016
Original
2016
Cited alongside, same era.
H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, “Pruning filters for efficient convnets,” arXiv preprint arXiv:1608.08710 , 2016
Original
2016
Cited alongside, same era.

Then

S. Han, J. Pool, S. Narang, H. Mao, S. Tang, E. Elsen, B. Catanzaro, J. Tran, and W. J. Dally, “DSD: Regularizing deep neural networks with dense-sparse-dense training flow,” arXiv preprint arXiv:1607.04381 , 2016
Original
2016
Later among the works it cites.
S. Anwar and W. Sung, “Compact deep convolutional neural networks with coarse pruning,” arXiv preprint arXiv:1610.09639 , 2016
Original
2016
Later among the works it cites.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin et al. , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” arXiv preprint arXiv:1603.04467 , 2016
Original
2016
Later among the works it cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition , 2016, pp. 770–778
2016
Later among the works it cites.
J. Hawkins, “Machines won’t become intelligent unless they incorporate certain features of the human brain,” IEEE Spectrum , vol. 54, no. 6, pp. 34–71, Jun. 2017
2017
Closest in time.
E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. V. Le, and A. Kurakin, “Large-scale evolution of image classifiers,” in Proc. Int. Conf. Machine Learning , 2017, pp. 2902–2911
2017
Closest in time.
R. Negrinho and G. Gordon, “Deeparchitect: Automatically designing and training deep architectures,” arXiv preprint arXiv:1704.08792 , 2017
Original
2017
Closest in time.
R. Miikkulainen, J. Liang, E. Meyerson, A. Rawal, D. Fink, O. Francon, B. Raju, H. Shahrzad, A. Navruzyan, N. Duffy et al. , “Evolving deep neural networks,” arXiv preprint arXiv:1703.00548 , 2017
Original
2017
Closest in time.
B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” arXiv preprint arXiv:1707.07012 , 2017
Original
2017
Closest in time.
X. Dong, S. Chen, and S. J. Pan, “Learning to prune deep neural networks via layer-wise optimal brain surgeon,” arXiv preprint arXiv:1705.07565 , 2017
Original
2017
Closest in time.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in PyTorch,” NIPS Workshop Autodiff , 2017
2017
Closest in time.
“PyTorch pre-trained VGG-16,” https://download.pytorch.org/models/vgg16-397923af.pth , 2017
2017
Closest in time.
G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten, “Densely connected convolutional networks,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition , vol. 1, no. 2, 2017, pp. 3–12
2017
Closest in time.
Y. Zhou and G. Diamos, “Neural architect: A multi-objective neural architecture search with performance prediction,” in Proc. Conf. SysML , 2018
2018
Closest in time.
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Proc. Advances in Neural Information Processing Systems , 2016, pp. 2074–2082
2082
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…