Fetching the paper…

Reading the bibliography…

2017

In-Place Activated BatchNorm for Memory-Optimized Training of DNNs

Bulò, Samuel Rota, Porzi, Lorenzo, Kontschieder, Peter

Original
PDF
Post on X

Understand

In this work we present In-Place Activated Batch Normalization (InPlace-ABN) - a novel approach to drastically reduce the training memory footprint of modern deep neural networks in a computationally efficient way.

Our solution substitutes the conventionally used succession of BatchNorm + Activation layers with a single plugin layer, hence avoiding invasive framework surgery while providing straightforward applicability for existing deep learning frameworks.
We obtain memory savings of up to 50% by dropping intermediate results and by recovering required information during the backward pass through the inversion of stored forward results, with only minor increase (0.8-2%) in computation time.
Also, we demonstrate how frequently used checkpointing approaches can be made computationally as efficient as InPlace-ABN.

In-Place Activated BatchNorm for Memory-Optimized Training of DNNs · Around

Built on

Training Deep and Recurrent Networks with Hessian-Free Optimization
J. Martens and I. Sutskever · 2012
Earlier work this paper cites.
Rectifier nonlinearities improve neural network acoustic models
A. L. Maas, A. Y. Hannun, and A. Y. Ng · 2013
Earlier work this paper cites.
Caffe: Convolutional architecture for fast feature embedding
Original
Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell · 2014
Earlier work this paper cites.
Microsoft COCO: Common objects in context
Original
T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
K. Simonyan and A. Zisserman · 2014
Earlier work this paper cites.
Binaryconnect: Training deep neural networks with binary weights during propagations
M. Courbariaux, Y. Bengio, and J.-P. David · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2015
Earlier work this paper cites.
Batch normalization: Accelerating deep network training by reducing internal covariate shift
Original
S. Ioffe and C. Szegedy · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karphathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei · 2015
Earlier work this paper cites.
Empirical evaluation of rectified activations in convolutional network
Original
B. Xu, N. Wang, T. Chen, and M. Li · 2015
Earlier work this paper cites.
COCO-Stuff: Thing and stuff classes in context
Original
H. Caesar, J. R. R. Uijlings, and V. Ferrari · 2016
Earlier work this paper cites.
Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs
Original
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille · 2016
Earlier work this paper cites.
Training deep nets with sublinear memory cost
Original
T. Chen, B. Xu, C. Zhang, and C. Guestrin · 2016
Earlier work this paper cites.

Similar

The Cityscapes dataset for semantic urban scene understanding
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele · 2016
Cited alongside, same era.
Memory-efficient backpropagation through time
A. Gruslys, R. Munos, I. Danihelka, M. Lanctot, and A. Graves · 2016
Cited alongside, same era.
Identity mappings in deep residual networks
Original
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.
Binarized neural networks
I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio · 2016
Cited alongside, same era.
Quantized neural networks: Training neural networks with low precision weights and activations
Original
I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio · 2016
Cited alongside, same era.

Then

Rethinking atrous convolution for semantic image segmentation
Original
L. Chen, G. Papandreou, F. Schroff, and H. Adam · 2017
Closest in time.
Deformable convolutional networks
Original
J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei · 2017
Closest in time.
Semantic video cnns through representation warping
Original
R. Gadde, V. Jampani, and P. V. Gehler · 2017
Closest in time.
The reversible residual network: Backpropagation without storing activations
A. N. Gomez, M. Ren, R. Urtasun, and R. B. Grosse · 2017
Closest in time.
Squeeze-and-excitation networks
Original
J. Hu, L. Shen, and G. Sun · 2017
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Inception-v4, inception-resnet and the impact of residual connections on learning

C. Szegedy, S. Ioffe, and V. Vanhoucke · 2016

Cited alongside, same era.

High-performance semantic segmentation using very deep fully convolutional networks

Z. Wu, C. Shen, and A. van den Hengel · 2016

Cited alongside, same era.

Wider or deeper: Revisiting the resnet model for visual recognition

Z. Wu, C. Shen, and A. van den Hengel · 2016

Cited alongside, same era.

Aggregated residual transformations for deep neural networks

S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He · 2016

Cited alongside, same era.

Wide residual networks

S. Zagoruyko and N. Komodakis · 2016

Cited alongside, same era.

Pyramid scene parsing network

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia · 2016

Cited alongside, same era.

Semantic understanding of scenes through the ADE20K dataset

B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba · 2016

Cited alongside, same era.

Densely connected convolutional networks

G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger · 2017

Closest in time.

Mixed precision training

P. Micikevicius, S. Narang, J. Alben, G. F. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu · 2017

Closest in time.

The mapillary vistas dataset for semantic understanding of street scenes

G. Neuhold, T. Ollmann, S. Rota Bulò, and P. Kontschieder · 2017

Closest in time.

Memory-efficient implementation of densenets

G. Pleiss, D. Chen, G. Huang, T. Li, L. van der Maaten, and K. Q. Weinberger · 2017

Closest in time.

Loss max-pooling for semantic image segmentation

S. Rota Bulò, G. Neuhold, and P. Kontschieder · 2017

Closest in time.

Understanding convolution for semantic segmentation

P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. W. Cottrell · 2017

Closest in time.

LSUN2017 segmentation challenge winning team PSPNet, July 2017

Y. Zhang, H. Zhao, and J. Shi · 2017

Closest in time.