2016

Systematic evaluation of CNN advances on the ImageNet

Mishkin, Dmytro, Sergievskiy, Nikolay, Matas, Jiri

Understand

The paper systematically studies the impact of a range of recent advances in CNN architectures and learning methods on the object categorization (ILSVRC) problem.

The evalution tests the influence of the following choices of the architecture: non-linearity (ReLU, ELU, maxout, compatibility with batch normalization), pooling variants (stochastic, max, average, mixed), network width, classifier design (convolutional, fully-connected, SPP), image pre-processing, and of learning parameters: learning rate, batch size, cleanliness of the data, etc.
The performance gains of the proposed modifications are first tested individually and then in combination.
The sum of individual gains is bigger than the observed improvement when all modifications are introduced, but the "deficit" is small suggesting independence of their benefits.

Built on

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15 (2014) 1929–1958
1958
Earlier work this paper cites.
doi:http://dx.doi.org/10.1016/S0146-664X(77)80011-7
R. Hummel, Image enhancement by histogram transformation, Computer Graphics and Image Processing 6 (2) (1977) 184 – 195 · 1977
Earlier work this paper cites.
D. E. Rumelhart, G. E. Hinton, R. J. Williams, Parallel distributed processing: Explorations in the microstructure of cognition, vol. 1, MIT Press, Cambridge, MA, USA, 1986, Ch. Learning Internal Representations by Error Propagation, pp. 318–362
1986
Earlier work this paper cites.
K. Zuiderveld, Graphics gems iv, Academic Press Professional, Inc., San Diego, CA, USA, 1994, Ch. Contrast Limited Adaptive Histogram Equalization, pp. 474–485
1994
Earlier work this paper cites.
Y. LeCun, L. Bottou, G. B. Orr, K.-R. Müller, Effiicient backprop, in: Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, Springer-Verlag, London, UK, UK, 1998, pp. 9–50
1998
Earlier work this paper cites.
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, in: Proceedings of the IEEE, Vol. 86, 1998, pp. 2278–2324
1998
Earlier work this paper cites.
doi:10.1016/S0893-6080(03)00138-2
D. R. Wilson, T. R. Martinez, The general inefficiency of batch training for gradient descent learning, Neural Netw. 16 (10) (2003) 1429–1451 · 2003
Earlier work this paper cites.
doi:10.1007/s11263-009-0275-4
M. Everingham, L. Gool, C. K. Williams, J. Winn, A. Zisserman, The pascal visual object classes (voc) challenge, International Journal of Computer Vision (IJCV) 88 (2) (2010) 303–338 · 2010
Earlier work this paper cites.

Similar

X. Glorot, A. Bordes, Y. Bengio, Deep sparse rectifier neural networks, in: G. J. Gordon, D. B. Dunson (Eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS-11), Vol. 15, Journal of Machine Learning Research - Workshop and Conference Proceedings, 2011, pp. 315–323
2011
Cited alongside, same era.
Y. Bengio, Neural Networks: Tricks of the Trade: Second Edition, Springer Berlin Heidelberg, Berlin, Heidelberg, 2012, Ch. Practical Recommendations for Gradient-Based Training of Deep Architectures, pp. 437–478
2012
Cited alongside, same era.
A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: F. Pereira, C. Burges, L. Bottou, K. Weinberger (Eds.), Advances in Neural Information Processing Systems 25, Curran Associates, Inc., 2012, pp. 1097–1105
2012
Cited alongside, same era.
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, Y. Bengio, Maxout networks, in: Proceedings of the 30th International Conference on Machine Learning, ICML 2013, Atlanta, GA, USA, 16-21 June 2013, 2013, pp. 1319–1327
2013
Cited alongside, same era.
B. Graham, Train you very own deep convolutional network (2014). URL https://www.kaggle.com/c/cifar-10/forums/t/10493/train-you-very-own-deep-convolutional-network
2014
Cited alongside, same era.
P. Swietojanski, J. Li, J.-T. Huang, Investigation of maxout networks for speech recognition, in: Proceedings of ICASSP, 2014
2014
Cited alongside, same era.
arXiv:1412.6806
Original
J. T. Springenberg, A. Dosovitskiy, T. Brox, M. Riedmiller, Striving for Simplicity: The All Convolutional Net, in: Proceedings of ICLR Workshop, 2014 · 2014
Cited alongside, same era.
Y. Taigman, M. Yang, M. Ranzato, L. Wolf, Deepface: Closing the gap to human-level performance in face verification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1701–1708
2014
Cited alongside, same era.
J. Dai, K. He, J. Sun, Instance-aware Semantic Segmentation via Multi-task Network Cascades, arXiv e-prints arXiv:1512.04412
Original
Cited in the paper.
G. Tolias, R. Sicre, H. Jégou, Particular object retrieval with integral max-pooling of CNN activations, arXiv e-prints arXiv:1511.05879
Original
Cited in the paper.
H. Nam, B. Han, Learning Multi-Domain Convolutional Neural Networks for Visual Tracking, arXiv e-prints arXiv:1510.07945
Original
Cited in the paper.
M. Jaderberg, K. Simonyan, A. Vedaldi, A. Zisserman, Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition, arXiv e-prints arXiv:1406.2227
Original
Cited in the paper.

Then

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision (IJCV) (2015) 1–42 doi:10.1007/s11263-015-0816-y
2015
Later among the works it cites.
K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale visual recognition, in: Proceedings of ICLR, 2015
2015
Later among the works it cites.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, Going deeper with convolutions, in: CVPR 2015, 2015
2015
Later among the works it cites.
S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: D. Blei, F. Bach (Eds.), Proceedings of the 32nd International Conference on Machine Learning (ICML-15), JMLR Workshop and Conference Proceedings, 2015, pp. 448–456
2015
Later among the works it cites.
B. Graham, Kaggle diabetic retinopathy detection competition report, Tech. rep. (2015)
2015
Later among the works it cites.
F. Schroff, D. Kalenichenko, J. Philbin, Facenet: A unified embedding for face recognition and clustering, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 815–823
2015
Later among the works it cites.
D. Mishkin, J. Matas, All you need is a good init, in: Proceedings of ICLR, 2016
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…