2020

Towards an Adversarially Robust Normalization Approach

Awais, Muhammad, Shamshad, Fahad, Bae, Sung-Ho

Understand

Batch Normalization (BatchNorm) is effective for improving the performance and accelerating the training of deep neural networks.

However, it has also shown to be a cause of adversarial vulnerability, i.e., networks without it are more robust to adversarial attacks.
In this paper, we investigate how BatchNorm causes this vulnerability and proposed new normalization that is robust to adversarial attacks.
We first observe that adversarial images tend to shift the distribution of BatchNorm input, and this shift makes train-time estimated population statistics inaccurate.

Built on

Adversarial examples improve image recognition
Original
Xie, C., Tan, M., Gong, B., Wang, J., Yuille, A., and Le, Q. V · 1911
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. et al · 2009
Earlier work this paper cites.
MNIST handwritten digit database
LeCun, Y. and Cortes, C · 2010
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Intriguing properties of neural networks
Original
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R · 2013
Earlier work this paper cites.
Explaining and harnessing adversarial examples
Original
Goodfellow, I. J., Shlens, J., and Szegedy, C · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Earlier work this paper cites.
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., and Clune, J · 2015
Earlier work this paper cites.
Layer normalization
Original
Ba, J. L., Kiros, J. R., and Hinton, G. E · 2016
Earlier work this paper cites.
Robustness of classifiers: from adversarial to random noise
Fawzi, A., Moosavi-Dezfooli, S.-M., and Frossard, P · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Earlier work this paper cites.
Deep networks with stochastic depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q · 2016
Earlier work this paper cites.
Adversarial machine learning at scale
Original
Kurakin, A., Goodfellow, I., and Bengio, S · 2016
Earlier work this paper cites.

Similar

Distillation as a defense to adversarial perturbations against deep neural networks
Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami, A · 2016
Cited alongside, same era.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Salimans, T. and Kingma, D. P · 2016
Cited alongside, same era.
Instance normalization: The missing ingredient for fast stylization
Original
Ulyanov, D., Vedaldi, A., and Lempitsky, V · 2016
Cited alongside, same era.
Wide residual networks
Original
Zagoruyko, S. and Komodakis, N · 2016
Cited alongside, same era.
Towards evaluating the robustness of neural networks
Carlini, N. and Wagner, D · 2017
Cited alongside, same era.
Densely connected convolutional networks
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q · 2017
Cited alongside, same era.
Batch renormalization: Towards reducing minibatch dependence in batch-normalized models
Ioffe, S · 2017
Cited alongside, same era.
Towards deep learning models resistant to adversarial attacks
Original
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A · 2017
Cited alongside, same era.
Ensemble adversarial training: Attacks and defenses
Original
Tramèr, F., Kurakin, A., Papernot, N., Goodfellow, I., Boneh, D., and McDaniel, P · 2017
Cited alongside, same era.
Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms
Original
Xiao, H., Rasul, K., and Vollgraf, R · 2017
Cited alongside, same era.
Aggregated residual transformations for deep neural networks
Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K · 2017
Cited alongside, same era.
Threat of adversarial attacks on deep learning in computer vision: A survey
Akhtar, N. and Mian, A · 2018
Cited alongside, same era.

Then

Adversarial spheres
Original
Gilmer, J., Metz, L., Faghri, F., Schoenholz, S. S., Raghu, M., Wattenberg, M., and Goodfellow, I · 2018
Later among the works it cites.
Norm matters: efficient and accurate normalization schemes in deep networks
Hoffer, E., Banner, R., Golan, I., and Soudry, D · 2018
Later among the works it cites.
Excessive invariance causes adversarial vulnerability
Original
Jacobsen, J.-H., Behrmann, J., Zemel, R., and Bethge, M · 2018
Later among the works it cites.
How does batch normalization help optimization?
Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A · 2018
Later among the works it cites.
Adversarially robust generalization requires more data
Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., and Madry, A · 2018
Later among the works it cites.
Adversarial vulnerability of neural networks increases with input dimension
Original
Simon-Gabriel, C.-J., Ollivier, Y., Bottou, L., Schölkopf, B., and Lopez-Paz, D · 2018
Later among the works it cites.
Group normalization
Wu, Y. and He, K · 2018
Later among the works it cites.
On the sensitivity of adversarial robustness to input data distributions
Ding, G. W., Lui, K. Y. C., Jin, X., Wang, L., and Huang, R · 2019
Later among the works it cites.
On the connection between adversarial robustness and saliency map interpretability
Original
Etmann, C., Lunz, S., Maass, P., and Schönlieb, C.-B · 2019
Later among the works it cites.
Adversarial examples are a natural consequence of test error in noise
Original
Ford, N., Gilmer, J., Carlini, N., and Cubuk, D · 2019
Later among the works it cites.
Batch normalization is a cause of adversarial vulnerability
Original
Galloway, A., Golubeva, A., Tanay, T., Moussa, M., and Taylor, G. W · 2019
Later among the works it cites.
Gpipe: Efficient training of giant neural networks using pipeline parallelism
Huang, Y., Cheng, Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al · 2019
Later among the works it cites.
Adversarial examples are not bugs, they are features
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A · 2019
Later among the works it cites.
Exploiting excessive invariance caused by norm-bounded adversarial robustness
Original
Jacobsen, J.-H., Behrmannn, J., Carlini, N., Tramer, F., and Papernot, N · 2019
Later among the works it cites.
Adversarial examples: Attacks and defenses for deep learning
Yuan, X., He, P., Zhu, Q., and Li, X · 2019
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…