Fetching the paper…

Reading the bibliography…

2016

Residual Networks of Residual Networks: Multilevel Residual Networks

Zhang, Ke, Sun, Miao, Han, Tony X. et al.

Original
PDF
Post on X

Understand

A residual-networks family with hundreds or even thousands of layers dominates major image recognition tasks, but building a network by simply stacking residual blocks inevitably limits its optimization ability.

This paper proposes a novel residual-network architecture, Residual networks of Residual networks (RoR), to dig the optimization ability of residual networks.
RoR substitutes optimizing residual mapping of residual mapping for optimizing original residual mapping.
In particular, RoR adds level-wise shortcut connections upon original residual networks to promote the learning capability of residual networks.

Residual Networks of Residual Networks: Multilevel Residual Networks · Around

Built on

S. Gross, and M. Wilber, “Training and investigating residual nets,” Facebook AI Research, CA. [Online]. Avilable:http://torch.ch/blog/2016/02/04/resnets.html , 2016
Original
2006
Earlier work this paper cites.
A. Krizhenvshky, and G. Hinton, “Learning multiple layers of features from tiny images,” M.Sc. thesis, Dept. of Comput. Sci., Univ. of Toronto, Toronto, ON, Canada, 2009
2009
Earlier work this paper cites.
X. Glorot, and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proc. Conf. Art. Intell. Stat. , 2010, pp. 249–256
2010
Earlier work this paper cites.
D. Erhan, Y. Bengio, A. Courville, P. A. Manzagol, P. Vincent, and S. Bengio, “Why does unsupervised pre-training help deep learning?,” The Journal of Machine Learning Research , vol. 11, pp. 625–660, Mar. 2010
2010
Earlier work this paper cites.
V. Nair, and G. Hinton, “Rectified linear units improve restricted Boltzmann machines,” in Proc. ICML , 2010, pp. 807–814
2010
Earlier work this paper cites.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in Proc. NIPS Workshop Deep Learning and Unsupervised feature learning. , 2011, pp. 1–9
2011
Earlier work this paper cites.
A. Krizhenvshky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional networks,” in Proc. Adv. Neural Inf. Process. Syst. , 2012, pp. 1097–1105
2012
Earlier work this paper cites.
G. Hinton, N. Srivastava, A. Krizhevsky, and K. Weinberger, “Improving neural networks by preventing co-adaptation of feature detectors,” arXiv preprint arXiv:1207.0580 , 2012
Original
2012
Earlier work this paper cites.
M. Lin, Q. Chen, and S. Yan, “Network in network,” arXiv preprint arXiv:1312.4400 , 2013
Original
2013
Earlier work this paper cites.
P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “Overfeat: Integrated recognition, localization and detection using convolutional networks,” arXiv preprint arXiv:1312.6229 , 2013
Original
2013
Earlier work this paper cites.
L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in Proc. ICML. , 2013, pp. 1058–1066
2013
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” arXiv preprint arXiv:1409.0575 , 2014
Original
2014
Earlier work this paper cites.
W. Y. Zou, X. Y. Wang, M. Sun, and Y. Lin, “Generic object detection with dense neural patterns and regional,” arXiv preprint arXiv:1404.4316 , 2014
Original
2014
Earlier work this paper cites.

Similar

K. Simonyan, and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014
Original
2014
Cited alongside, same era.
A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: hints for thin deep nets,” arXiv preprint arXiv:1412.6550 , 2014
Original
2014
Cited alongside, same era.
J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” arXiv preprint arXiv:1412.6806 , 2014
Original
2014
Cited alongside, same era.
Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. Neural Networks , vol. 5, no. 2, pp. 157–166, Aug. 2014
2014
Cited alongside, same era.

Then

K. He, and J. Sun, “Convolutional neural networks at constrained time cost,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2015, pp. 5353–5360
2015
Later among the works it cites.
R. K. Srivastava, K. Greff, and J. Schmidhuber, “Highway networks,” arXiv preprint arXiv:1505.00387 , 2015
Original
2015
Later among the works it cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Identity mapping in deep residual networks,” arXiv preprint arXiv:1603.05027 , 2016
Original
2016
Closest in time.
S. Zagoruyko, and N. Komodakis, “Wide residual networks,” arXiv preprint arXiv:1605.07146 , 2016
Original
2016
Closest in time.
G. Huang, Y. Sun, Z. Liu, and K. Weinberger, “Deep networks with stochastic depth,” arXiv preprint arXiv:1605.09382 , 2016

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research , vol. 15, pp. 1929–1958, Jun. 2014

2014

Cited alongside, same era.

Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature , vol. 521, no. 7553, pp. 436–444, May. 2015

2015

Cited alongside, same era.

C. -Y. Lee, S. Xie, P. Gallagher, Z. Zhang, and Z. Tu, “Deeply-supervised nets,” in Proc. AISTATS , 2015, pp. 562–570

2015

Cited alongside, same era.

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recog. , 2015, pp. 1–9

2015

Cited alongside, same era.

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” arXiv preprint arXiv:1512.03385 , 2015

2015

Cited alongside, same era.

D. -A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” arXiv preprint arXiv:1511.07289 , 2015

2015

Cited alongside, same era.

K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” arXiv preprint arXiv:1502.01852 , 2015

2015

Cited alongside, same era.

S. Ioffe, and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167 , 2015

2015

Cited alongside, same era.

2016

Closest in time.

L. Trottier, P. Giguere, and B. Chaib-draa, “Parametric exponential linear unit for deep convolutional neural networks,” arXiv preprint arXiv:1605.09322 , 2016

2016

Closest in time.

G. Larsson, M. Maire, and G. Shakhnarovich, “FractalNet: ultra-deep neural networks without residuals,” arXiv preprint arXiv:1605.07648 , 2016

2016

Closest in time.

S. Targ, D. Almeida, and K. Lyman, “Resnet in resnet: generalizing residual architectures,” arXiv preprint arXiv:1603.08029 , 2016

2016

Closest in time.

A. Shah, S. Shinde, E. Kadam, and H. Shah, “Deep residual networks with exponential linear unit,” arXiv preprint arXiv:1604.04112 , 2016

2016

Closest in time.

S. Singh, D. Hoiem, and D. Forsyth, “Swapout: learning an ensemble of deep architectures,” arXiv preprint arXiv:1605.06465 , 2016

2016

Closest in time.

F. Shen, and G. Zeng, “Weighted residuals for very deep networks,” arXiv preprint arXiv:1605.08831 , 2016

2016

Closest in time.

J. Moniz, and C. Pal, “Convolutional residual memory networks,” arXiv preprint arXiv:1606.05262 , 2016

2016

Closest in time.