2021

Rescaling CNN through Learnable Repetition of Network Parameters

Chavan, Arnav, Bamba, Udbhav, Tiwari, Rishabh et al.

Understand

Deeper and wider CNNs are known to provide improved performance for deep learning tasks.

However, most such networks have poor performance gain per parameter increase.
In this paper, we investigate whether the gain observed in deeper models is purely due to the addition of more optimization parameters or whether the physical size of the network as well plays a role.
Further, we present a novel rescaling strategy for CNNs based on learnable repetition of its parameters.

An empirical evaluation of deep architectures on problems with many factors of variation
Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio · 2007
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky · 2009
Earlier work this paper cites.
The pascal visual object classes (voc) challenge
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman · 2010
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Xavier Glorot and Yoshua Bengio · 2010
Earlier work this paper cites.
Learning both weights and connections for efficient neural networks
Original
Song Han, Jeff Pool, John Tran, and William J Dally · 2015
Earlier work this paper cites.
U-net: Convolutional networks for biomedical image segmentation
O. Ronneberger, P. Fischer, and T. Brox · 2015
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
K. Simonyan and Andrew Zisserman · 2015
Earlier work this paper cites.
Data-free parameter pruning for deep neural networks
Original
Suraj Srinivas and R Venkatesh Babu · 2015
Earlier work this paper cites.
Group equivariant convolutional networks
Taco Cohen and Max Welling · 2016
Earlier work this paper cites.

Steerable cnns
Original
Taco S Cohen and Max Welling · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Wide residual networks
Original
Sergey Zagoruyko and Nikos Komodakis · 2016
Cited alongside, same era.
Sharesnet: reducing residual network parameter number by sharing weights
Original
Alexandre Boulch · 2017
Cited alongside, same era.
Learning efficient convolutional networks through network slimming
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang · 2017
Cited alongside, same era.
Learning sparse neural networks through
Original
Christos Louizos, Max Welling, and Diederik P Kingma · 2017
Cited alongside, same era.
Variational dropout sparsifies deep neural networks
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov · 2017
Cited alongside, same era.
Searching for activation functions
Original
Prajit Ramachandran, Barret Zoph, and Quoc V Le · 2017
Cited alongside, same era.
Compressing neural networks using the variational information bottleneck
Bin Dai, Chen Zhu, Baining Guo, and David Wipf · 2018
Cited alongside, same era.

The lottery ticket hypothesis: Finding sparse, trainable neural networks
Original
Jonathan Frankle and Michael Carbin · 2018
Later among the works it cites.
Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng · 2018
Later among the works it cites.
Learning steerable filters for rotation equivariant cnns
M. Weiler, F. A. Hamprecht, and M. Storath · 2018
Later among the works it cites.
Deep anchored convolutional neural networks
Jiahui Huang, Kshitij Dwivedi, and Gemma Roig · 2019
Later among the works it cites.
Structured pruning of neural networks with budget-aware regularization
Carl Lemaire, Andrew Achkar, and Pierre-Marc Jodoin · 2019
Later among the works it cites.
EfficientNet: Rethinking model scaling for convolutional neural networks
Mingxing Tan and Quoc Le · 2019
Later among the works it cites.
General E(2)-Equivariant Steerable CNNs
Maurice Weiler and Gabriele Cesa · 2019
Later among the works it cites.
Training highly effective connectivities within neural networks with randomly initialized, fixed weights
Original
Cristian Ivan and R. Florian · 2020
Later among the works it cites.
Chipnet: Budget-aware pruning with heaviside continuous approximations
Rishabh Tiwari, Udbhav Bamba, Arnav Chavan, and Deepak Gupta · 2021
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…