2020

Squared $\ell_2$ Norm as Consistency Loss for Leveraging Augmented Data to Learn Robust and Invariant Representations

Wang, Haohan, Huang, Zeyi, Wu, Xindi et al.

Understand

Data augmentation is one of the most popular techniques for improving the robustness of neural networks.

In addition to directly training the model with original samples and augmented samples, a torrent of methods regularizing the distance between embeddings/representations of the original samples and their augmented counterparts have been introduced.
In this paper, we explore these various regularization choices, seeking to provide a general understanding of how we should regularize the embeddings.
Our analysis suggests the ideal choices of regularization correspond to various assumptions.

Built on

Gradient-based learning applied to document recognition
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner · 1998
Earlier work this paper cites.
Introduction to statistical learning theory
O. Bousquet, S. Boucheron, and G. Lugosi · 2003
Earlier work this paper cites.
Topics in optimal transportation
C. Villani · 2003
Earlier work this paper cites.
Optimal transport: old and new , volume 338
C. Villani · 2008
Earlier work this paper cites.
A theory of learning from different domains
S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan · 2010
Earlier work this paper cites.
Intriguing properties of neural networks
Original
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus · 2013
Earlier work this paper cites.
Fast computation of wasserstein barycenters
M. Cuturi and A. Doucet · 2014
Earlier work this paper cites.
Spatial transformer networks
M. Jaderberg, K. Simonyan, A. Zisserman, et al · 2015
Earlier work this paper cites.
Hyper-class augmented and regularized deep learning for fine-grained image classification
S. Xie, T. Yang, X. Wang, and Y. Lin · 2015
Earlier work this paper cites.
Group equivariant convolutional networks
T. Cohen and M. Welling · 2016
Earlier work this paper cites.
Adaptive data augmentation for image classification
A. Fawzi, H. Samulowitz, D. S. Turaga, and P. Frossard · 2016
Earlier work this paper cites.
Domain-adversarial training of neural networks
Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky · 2016
Earlier work this paper cites.
Nips 2016 tutorial: Generative adversarial networks
Original
I. Goodfellow · 2016
Earlier work this paper cites.
Cs229t/stat231: Statistical learning theory (winter 2016), 2016
P. Liang · 2016
Earlier work this paper cites.
Regularization with stochastic transformations and perturbations for deep semi-supervised learning
M. Sajjadi, M. Javanmardi, and T. Tasdizen · 2016
Earlier work this paper cites.
Improving the robustness of deep neural networks via stability training
S. Zheng, Y. Song, T. Leung, and I. Goodfellow · 2016
Earlier work this paper cites.
Wasserstein gan, 2017
M. Arjovsky, S. Chintala, and L. Bottou · 2017
Earlier work this paper cites.

Similar

Improved training of wasserstein gans, 2017
I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville · 2017
Cited alongside, same era.
Information dropout: Learning optimal representations through noisy computation
A. Achille and S. Soatto · 2018
Cited alongside, same era.
Data augmentation instead of explicit regularization, 2018
A. Hernández-García and P. König · 2018
Cited alongside, same era.
Adversarial logit pairing, 2018
H. Kannan, A. Kurakin, and I. Goodfellow · 2018
Cited alongside, same era.
Clebsch–gordan nets: a fully fourier space spherical convolutional neural network
R. Kondor, Z. Lin, and S. Trivedi · 2018
Cited alongside, same era.
Learning noise-invariant representations for robust speech recognition
D. Liang, Z. Huang, and Z. C. Lipton · 2018
Cited alongside, same era.
Towards deep learning models resistant to adversarial attacks
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu · 2018
Cited alongside, same era.
One-dimensional empirical measures, order statistics, and Kantorovich transport distances , volume 261
S. Bobkov and M. Ledoux · 2019
Cited alongside, same era.
A group-theoretic framework for data augmentation, 2019
S. Chen, E. Dobriban, and J. H. Lee · 2019
Cited alongside, same era.
Autoaugment: Learning augmentation strategies from data
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le · 2019
Cited alongside, same era.
A kernel theory of modern data augmentation
T. Dao, A. Gu, A. Ratner, V. Smith, C. D. Sa, and C. Ré · 2019
Cited alongside, same era.
Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel · 2019
Cited alongside, same era.

Then

Improving robustness without sacrificing accuracy with patch gaussian augmentation
Original
R. G. Lopes, D. Yin, B. Poole, J. Gilmer, and E. D. Cubuk · 2019
Later among the works it cites.
Does data augmentation lead to positive margin?
S. Rajput, Z. Feng, Z. Charles, P.-L. Loh, and D. Papailiopoulos · 2019
Later among the works it cites.
Cycle-consistency for robust visual question answering
M. Shah, X. Chen, M. Rohrbach, and D. Parikh · 2019
Later among the works it cites.
A survey on image data augmentation for deep learning
C. Shorten and T. M. Khoshgoftaar · 2019
Later among the works it cites.
Equivariant transformer networks
Original
K. S. Tai, P. Bailis, and G. Valiant · 2019
Later among the works it cites.
Theoretical analysis of adversarial learning: A minimax approach
Z. Tu, J. Zhang, and D. Tao · 2019
Later among the works it cites.
Learning robust representations by projecting superficial statistics out
H. Wang, Z. He, Z. C. Lipton, and E. P. Xing · 2019
Later among the works it cites.
Regularized adversarial training (RAT) for robust cellular electron cryo tomograms classification
X. Wu, Y. Mao, H. Wang, X. Zeng, X. Gao, E. P. Xing, and M. Xu · 2019
Later among the works it cites.
Unsupervised data augmentation
Original
Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le · 2019
Later among the works it cites.
Invariance-inducing regularization using worst-case transformations suffices to boost accuracy and spatial robustness
F. Yang, Z. Wang, and C. Heinze-Deml · 2019
Later among the works it cites.
Theoretically principled trade-off between robustness and accuracy
H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan · 2019
Later among the works it cites.
Making convolutional networks shift-invariant again
Original
R. Zhang · 2019
Later among the works it cites.
Learning data augmentation strategies for object detection, 2019
Original
B. Zoph, E. D. Cubuk, G. Ghiasi, T.-Y. Lin, J. Shlens, and Q. V. Le · 2019
Later among the works it cites.
Logic-guided data augmentation and regularization for consistent question answering, 2020
A. Asai and H. Hajishirzi · 2020
Closest in time.
Augmix: A simple data processing method to improve robustness and uncertainty
D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan · 2020
Closest in time.
High frequency component helps explain the generalization of convolutional neural networks
H. Wang, X. Wu, Z. Huang, and E. P. Xing · 2020
Closest in time.
Adversarial autoaugment
X. Zhang, Q. Wang, J. Zhang, and Z. Zhong · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…