2016

Deep Residual Networks with Exponential Linear Unit

Shah, Anish, Kadam, Eashan, Shah, Hena et al.

Understand

Very deep convolutional neural networks introduced new problems like vanishing gradient and degradation.

The recent successful contributions towards solving these problems are Residual and Highway Networks.
These networks introduce skip connections that allow the information (from the input or those learned in earlier layers) to flow more into the deeper layers.
These very deep models have lead to a considerable decrease in test errors, on benchmarks like ImageNet and COCO.

Built on

LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., Jackel, L. D.: Backpropagation applied to handwritten zip code recognition. Neural computation, 1989
1989
Earlier work this paper cites.
LeCun, Y., Bottou, L., Orr, G. B., Muller, K.R.: Efficient backprop. ¨ In Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998
1998
Earlier work this paper cites.
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images (2009)
2009
Earlier work this paper cites.
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE (2009) 248–255
2009
Earlier work this paper cites.
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010
2010
Earlier work this paper cites.
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10). (2010) 807-814
2010
Earlier work this paper cites.
Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In NIPS, 2012
2012
Earlier work this paper cites.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., LeCun, Y.: Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229 (2013)
Original
2013
Earlier work this paper cites.
Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In Dasgupta, S., Mcallester, D., eds.: Proceedings of the 30th International Conference on Machine Learning (ICML- 13). Volume 28., JMLR Workshop and Conference Proceedings (May 2013) 1058–1066
2013
Earlier work this paper cites.
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint arXiv:1302.4389 (2013)
Original
2013
Earlier work this paper cites.

Similar

Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
Original
2013
Cited alongside, same era.
Simonyan, K., Zisserman, A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Original
2014
Cited alongside, same era.
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
Original
2014
Cited alongside, same era.
Lee, C.-Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. arXiv:1409.5185, 2014
Original
2014
Cited alongside, same era.
Graham, B.: Fractional max-pooling. arXiv preprint arXiv:1412.6071 (2014)
Original
2014
Cited alongside, same era.
Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)
Original
2014
Cited alongside, same era.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (2015)
Original
2015
Cited alongside, same era.
He, K., J. Sun, J.: Convolutional neural networks at constrained time cost. In CVPR, 2015
2015
Cited alongside, same era.
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015
2015
Cited alongside, same era.

Then

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015
2015
Later among the works it cites.
Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)
Original
2015
Later among the works it cites.
Clevert, D., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by Exponential Linear Units (ELUs). arXiv preprint arXiv:1511.07289 (2015)
Original
2015
Later among the works it cites.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 1–9
2015
Later among the works it cites.
Liang, M., Hu, X.: Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (2015) 3367–3375
2015
Later among the works it cites.
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Ali, M., Adams, R.P., et al.: Scalable bayesian optimization using deep neural networks. arXiv preprint arXiv:1502.05700 (2015)
Original
2015
Later among the works it cites.
Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems. (2015) 2368–2376
2015
Later among the works it cites.
Lee, C.Y., Gallagher, P.W., Tu, Z.: Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv:1509.08985 (2015)
Original
2015
Later among the works it cites.
K. He, X. Zhang, S. Ren, and J. Sun. Identity Mappings in Deep Residual Networks. arXiv preprint arXiv:1603.05027 (2016)
Original
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…