Built on

Gumbel, E.J.: Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series 33
1954
Earlier work this paper cites.
Mozer, M.C., Smolensky, P.: Skeletonization: A technique for trimming the fat from a network via relevance assessment. In: Advances in neural information processing systems. pp. 107–115 (1989)
1989
Earlier work this paper cites.
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in neural information processing systems. pp. 598–605 (1990)
1990
Earlier work this paper cites.
Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: Optimal brain surgeon. In: Advances in neural information processing systems. pp. 164–171 (1993)
1993
Earlier work this paper cites.
Viola, P., Jones, M.J.: Robust real-time face detection. International Journal of Computer Vision 57
2004
Earlier work this paper cites.
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. Ieee (2009)
2009
Earlier work this paper cites.
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012), http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
2012
Earlier work this paper cites.
Bengio, Y.: Deep learning of representations: Looking forward. CoRR abs/1305.0445
Original
2013
Earlier work this paper cites.
Bengio, Y., Léonard, N., Courville, A.C.: Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR abs/1308.3432
Original
2013
Earlier work this paper cites.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556
Original
2014
Earlier work this paper cites.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15
2014
Earlier work this paper cites.
Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: NIPS. pp. 3123–3131 (2015)
2015
Earlier work this paper cites.
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Original
2015
Earlier work this paper cites.
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems. pp. 1135–1143 (2015)
2015
Earlier work this paper cites.
Kingma, D.P., Salimans, T., Welling, M.: Variational dropout and the local reparameterization trick. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) NIPS, pp. 2575–2583. Curran Associates, Inc. (2015), http://papers.nips.cc/paper/5666-variational-dropout-and-the-local-reparameterization-trick.pdf
2015
Earlier work this paper cites.
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521
2015
Earlier work this paper cites.
Li, H., Lin, Z., Shen, X., Brandt, J., Hua, G.: A convolutional neural network cascade for face detection. In: CVPR. pp. 5325–5334 (2015)
2015
Earlier work this paper cites.
Graves, A.: Adaptive computation time for recurrent neural networks. CoRR abs/1603.08983
Original
2016
Earlier work this paper cites.

Similar

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
2016
Cited alongside, same era.
Hu, H., Peng, R., Tai, Y.W., Tang, C.K.: Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 (2016)
Original
2016
Cited alongside, same era.
Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: ECCV. pp. 646–661. Springer (2016)
2016
Cited alongside, same era.
Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)
Original
2016
Cited alongside, same era.
Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. ICLR (2016), arXiv preprint arXiv:1611.06440

Then

McGill, M., Perona, P.: Deciding how to decide: Dynamic routing in artificial neural networks. In: ICML (2017), arXiv preprint arXiv:1703.06217
Original
2017
Later among the works it cites.
Molchanov, D., Ashukha, A., Vetrov, D.: Variational dropout sparsifies deep neural networks. In: Precup, D., Teh, Y.W. (eds.) ICML. vol. 70, pp. 2498–2507. PMLR (06–11 Aug 2017), http://proceedings.mlr.press/v70/molchanov17a.html
2017
Later among the works it cites.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)
2017
Later among the works it cites.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)
Original
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2018

Channel selection using Gumbel Softmax

Herrmann, Charles, Bowen, Richard Strong, Zabih, Ramin

Understand

Important applications such as mobile computing require reducing the computational costs of neural network inference.

Ideally, applications would specify their preferred tradeoff between accuracy and speed, and the network would optimize this end-to-end, using classification error to remove parts of the network.
Increasing speed can be done either during training - e.g., pruning filters - or during inference - e.g., conditionally executing a subset of the layers.
We propose a single end-to-end framework that can improve inference efficiency in both settings.

Built on

Gumbel, E.J.: Statistical theory of extreme values and some practical applications. NBS Applied Mathematics Series 33
1954
Earlier work this paper cites.
Mozer, M.C., Smolensky, P.: Skeletonization: A technique for trimming the fat from a network via relevance assessment. In: Advances in neural information processing systems. pp. 107–115 (1989)
1989
Earlier work this paper cites.
LeCun, Y., Denker, J.S., Solla, S.A.: Optimal brain damage. In: Advances in neural information processing systems. pp. 598–605 (1990)
1990
Earlier work this paper cites.
Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: Optimal brain surgeon. In: Advances in neural information processing systems. pp. 164–171 (1993)
1993
Earlier work this paper cites.
Viola, P., Jones, M.J.: Robust real-time face detection. International Journal of Computer Vision 57
2004
Earlier work this paper cites.
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255. Ieee (2009)
2009
Earlier work this paper cites.
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012), http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
2012
Earlier work this paper cites.
Bengio, Y.: Deep learning of representations: Looking forward. CoRR abs/1305.0445
Original
2013
Earlier work this paper cites.
Bengio, Y., Léonard, N., Courville, A.C.: Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR abs/1308.3432
Original
2013
Earlier work this paper cites.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556
Original
2014
Earlier work this paper cites.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15
2014
Earlier work this paper cites.
Courbariaux, M., Bengio, Y., David, J.P.: Binaryconnect: Training deep neural networks with binary weights during propagations. In: NIPS. pp. 3123–3131 (2015)
2015
Earlier work this paper cites.
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Original
2015
Earlier work this paper cites.
Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems. pp. 1135–1143 (2015)
2015
Earlier work this paper cites.
Kingma, D.P., Salimans, T., Welling, M.: Variational dropout and the local reparameterization trick. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) NIPS, pp. 2575–2583. Curran Associates, Inc. (2015), http://papers.nips.cc/paper/5666-variational-dropout-and-the-local-reparameterization-trick.pdf
2015
Earlier work this paper cites.
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521
2015
Earlier work this paper cites.
Li, H., Lin, Z., Shen, X., Brandt, J., Hua, G.: A convolutional neural network cascade for face detection. In: CVPR. pp. 5325–5334 (2015)
2015
Earlier work this paper cites.
Graves, A.: Adaptive computation time for recurrent neural networks. CoRR abs/1603.08983
Original
2016
Earlier work this paper cites.

Similar

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
2016
Cited alongside, same era.
Hu, H., Peng, R., Tai, Y.W., Tang, C.K.: Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. arXiv preprint arXiv:1607.03250 (2016)
Original
2016
Cited alongside, same era.
Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: ECCV. pp. 646–661. Springer (2016)
2016
Cited alongside, same era.
Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)
Original
2016
Cited alongside, same era.
Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. ICLR (2016), arXiv preprint arXiv:1611.06440

Then

McGill, M., Perona, P.: Deciding how to decide: Dynamic routing in artificial neural networks. In: ICML (2017), arXiv preprint arXiv:1703.06217
Original
2017
Later among the works it cites.
Molchanov, D., Ashukha, A., Vetrov, D.: Variational dropout sparsifies deep neural networks. In: Precup, D., Teh, Y.W. (eds.) ICML. vol. 70, pp. 2498–2507. PMLR (06–11 Aug 2017), http://proceedings.mlr.press/v70/molchanov17a.html
2017
Later among the works it cites.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch. In: NIPS-W (2017)
2017
Later among the works it cites.
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., Dean, J.: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538 (2017)
Original
2017
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Original

2016

Cited alongside, same era.

Srinivas, S., Babu, R.V.: Generalized dropout. arXiv preprint arXiv:1611.06791 (2016)

Original

2016

Cited alongside, same era.

Srinivas, S., Babu, V.: Learning neural network architectures using backpropagation. In: BMVC. pp. 104.1–104.11 (September 2016). https://doi.org/10.5244/C.30.104, https://dx.doi.org/10.5244/C.30.104

2016

Cited alongside, same era.

Teerapittayanon, S., McDanel, B., Kung, H.: Branchynet: Fast inference via early exiting from deep neural networks. In: ICPR. pp. 2464–2469. IEEE (2016)

2016

Cited alongside, same era.

Yang, F., Choi, W., Lin, Y.: Exploit all the layers: Fast and accurate CNN object detector with scale dependent pooling and cascaded rejection classifiers. In: CVPR. pp. 2129–2137 (2016)

2016

Cited alongside, same era.

Bolukbasi, T., Wang, J., Dekel, O., Saligrama, V.: Adaptive neural networks for efficient inference. In: ICML. pp. 527–536 (2017)

2017

Cited alongside, same era.

Figurnov, M., Collins, M.D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D.P., Salakhutdinov, R.: Spatially adaptive computation time for residual networks. In: CVPR (2017)

2017

Cited alongside, same era.

Gal, Y., Hron, J., Kendall, A.: Concrete dropout. In: Advances in Neural Information Processing Systems. pp. 3581–3590 (2017)

2017

Cited alongside, same era.

Srinivas, S., Subramanya, A., Venkatesh Babu, R.: Training sparse neural networks. In: CVPR Workshops (July 2017)

2017

Later among the works it cites.

Veit, A., Belongie, S.: Convolutional networks with adaptive inference graphs. In: ECCV (2017), https://github.com/andreasveit/convnet-aig

2017

Later among the works it cites.

Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017)

Original

2017

Later among the works it cites.

Cai, H., Zhu, L., Han, S.: ProxylessNAS: Direct neural architecture search on target task and hardware. arXiv:1812.00332 (2018)

Original

2018

Closest in time.

He, Y., Lin, J., Liu, Z., Wang, H., Li, L.J., Han, S.: AMC: Automl for model compression and acceleration on mobile devices. In: ECCV. pp. 784–800 (2018)

2018

Closest in time.

Huang, Z., Wang, N.: Data-driven sparse structure selection for deep neural networks. ECCV (2018)

2018

Closest in time.

Luo, J.H., Wu, J.: Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. arXiv preprint arXiv:1805.08941 (2018)

Original

2018

Closest in time.

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: CVPR. pp. 4510–4520 (2018)

2018

Closest in time.

Shirakawa, S., Iwata, Y., Akimoto, Y.: Dynamic optimization of neural network structures using probabilistic modeling. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

2018

Closest in time.

Suau, X., Zappella, L., Palakkode, V., Apostoloff, N.: Principal filter analysis for guided network compression. arXiv preprint arXiv:1807.10585 (2018)

Original

2018

Closest in time.

Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: CVPR. pp. 8697–8710 (2018)

Original

2018

Closest in time.

He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4340–4349 (2019)

2019

Closest in time.

Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. In: ICLR (2019), {https://openreview.net/forum?id=rJlnB3C5Ym}

2019

Closest in time.

You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In: Advances in Neural Information Processing Systems. pp. 2130–2141 (2019)

2019

Closest in time.