2019

A Group-Theoretic Framework for Data Augmentation

Chen, Shuxiao, Dobriban, Edgar, Lee, Jane H

Understand

Data augmentation is a widely used trick when training deep neural networks: in addition to the original data, properly transformed data are also added to the training set.

However, to the best of our knowledge, a clear mathematical framework to explain the performance benefits of data augmentation is not available.
In this paper, we develop such a theoretical framework.
We show data augmentation is equivalent to an averaging operation over the orbits of a certain group that keeps the data distribution approximately invariant.

Built on

Determination of macromolecular structure in solution by spatial correlation of scattering fluctuations
Z. Kam · 1977
Earlier work this paper cites.
Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position
K. Fukushima · 1980
Earlier work this paper cites.
The reconstruction of structure from electron micrographs of randomly oriented particles
Z. Kam · 1980
Earlier work this paper cites.
An image synthesizer
K. Perlin · 1985
Earlier work this paper cites.
The calculation of posterior distributions by data augmentation
M. A. Tanner and W. H. Wong · 1987
Earlier work this paper cites.
Backpropagation applied to handwritten zip code recognition
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel · 1989
Earlier work this paper cites.
Document image defect models
H. S. Baird · 1992
Earlier work this paper cites.
Statistical analysis of spherical data
N. I. Fisher, T. Lewis, and B. J. Embleton · 1993
Earlier work this paper cites.
Group invariance in statistical inference
N. C. Giri · 1996
Earlier work this paper cites.
Theory of point estimation
E. Lehmann and G. Casella · 1998
Earlier work this paper cites.
Asymptotic statistics
A. W. Van der Vaart · 1998
Earlier work this paper cites.
Rademacher and gaussian complexities: Risk bounds and structural results
P. L. Bartlett and S. Mendelson · 2002
Earlier work this paper cites.
Topics in optimal transportation
C. Villani · 2003
Earlier work this paper cites.
Statistical inference under symmetry
I. S. Helland · 2004
Earlier work this paper cites.
Testing statistical hypotheses
E. L. Lehmann and J. P. Romano · 2005
Earlier work this paper cites.
Three-dimensional electron microscopy of macromolecular assemblies: visualization of biological molecules in their native state
J. Frank · 2006
Earlier work this paper cites.
Robust truncated hinge loss support vector machines
Y. Wu and Y. Liu · 2007
Earlier work this paper cites.
Time series: theory and methods
P. J. Brockwell and R. A. Davis · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky · 2009
Earlier work this paper cites.
Structure of isolated biomolecules obtained from ultrashort x-ray pulses: exploiting the symmetry of random orientations
D. K. Saldin, V. L. Shneerson, R. Fung, and A. Ourmazd · 2009
Earlier work this paper cites.
Variance components , volume 391
S. R. Searle, G. Casella, and C. E. McCulloch · 2009
Earlier work this paper cites.
Deep, big, simple neural nets for handwritten digit recognition
D. C. Cireşan, U. Meier, L. M. Gambardella, and J. Schmidhuber · 2010
Earlier work this paper cites.
Deep learners benefit more from out-of-distribution examples
Y. Bengio, F. Bastien, A. Bergeron, N. Boulanger-Lewandowski, T. Breuel, Y. Chherawala, M. Cisse, M. Côté, D. Erhan, J. Eustache, et al · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
A. Krizhevsky, I. Sutskever, and G. E. Hinton · 2012
Earlier work this paper cites.
Group invariant scattering
S. Mallat · 2012
Earlier work this paper cites.
Topics in random matrix theory
T. Tao · 2012
Earlier work this paper cites.
Invariant scattering convolution networks
J. Bruna and S. Mallat · 2013
Earlier work this paper cites.
Vocal tract length perturbation (vtlp) improves speech recognition
N. Jaitly and G. E. Hinton · 2013
Earlier work this paper cites.
Learning with marginalized corrupted features
L. Maaten, M. Chen, S. Tyree, and K. Weinberger · 2013
Earlier work this paper cites.
Algorithms for direct 0–1 loss optimization in binary classification
T. Nguyen and S. Sanner · 2013
Earlier work this paper cites.

Similar

Monte Carlo statistical methods
C. Robert and G. Casella · 2013
Cited alongside, same era.
Intriguing properties of neural networks
Original
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus · 2013
Cited alongside, same era.
Deep symmetry networks
R. Gens and P. M. Domingos · 2014
Cited alongside, same era.
Conditional generative adversarial nets
Original
M. Mirza and S. Osindero · 2014
Cited alongside, same era.
Understanding machine learning: From theory to algorithms
S. Shalev-Shwartz and S. Ben-David · 2014
Cited alongside, same era.
XTOP: high-resolution X-ray diffraction and imaging
V. Favre-Nicolin, J. Baruchel, H. Renevier, J. Eymery, and A. Borbély · 2015
Cited alongside, same era.
Simulations on time-resolved structure determination of uncrystallized biomolecules in the presence of shot noise
K. Pande, M. Schmidt, P. Schwander, and D. Saldin · 2015
Cited alongside, same era.
Deep learning in neural networks: An overview
J. Schmidhuber · 2015
Cited alongside, same era.
Steerable pca for rotation-invariant image recognition
C. Vonesch, F. Stauber, and M. Unser · 2015
Cited alongside, same era.
Exploiting cyclic symmetry in convolutional neural networks
Original
S. Dieleman, J. De Fauw, and K. Kavukcuoglu · 2016
Cited alongside, same era.
Dreaming more data: Class-dependent distributions over diffeomorphisms for learned data augmentation
S. Hauberg, O. Freifeld, A. B. L. Larsen, J. Fisher, and L. Hansen · 2016
Cited alongside, same era.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Cited alongside, same era.

Then

On the generalization of equivariance and convolution in neural networks to the action of compact groups
Original
R. Kondor and S. Trivedi · 2018
Later among the works it cites.
Clebsch–gordan nets: a fully fourier space spherical convolutional neural network
R. Kondor, Z. Lin, and S. Trivedi · 2018
Later among the works it cites.
e e pca: High dimensional exponential family pca
L. T. Liu, E. Dobriban, and A. Singer · 2018
Later among the works it cites.
Mathematics for cryo-electron microscopy
Original
A. Singer · 2018
Later among the works it cites.
Rendergan: Generating realistic labeled data
L. Sixt, B. Wild, and T. Landgraf · 2018
Later among the works it cites.
3d steerable cnns: Learning rotationally equivariant features in volumetric data
M. Weiler, M. Geiger, M. Welling, W. Boomsma, and T. Cohen · 2018
Later among the works it cites.
A mathematical theory of deep convolutional neural networks for feature extraction
T. Wiatowski and H. Bölcskei · 2018
Later among the works it cites.
Steerable e e pca
Original
Z. Zhao, L. T. Liu, and A. Singer · 2018
Later among the works it cites.
Symmetry-adapted representation learning
F. Anselmi, G. Evangelopoulos, L. Rosasco, and T. Poggio · 2019
Closest in time.
On exact computation with an infinitely wide neural net
S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang · 2019
Closest in time.
Probabilistic symmetry and invariant neural networks
Original
B. Bloem-Reddy and Y. W. Teh · 2019
Closest in time.
How much over-parameterization is sufficient to learn deep relu networks?
Original
Z. Chen, Y. Cao, D. Zou, and Q. Gu · 2019
Closest in time.
Certified adversarial robustness via randomized smoothing
Original
J. M. Cohen, E. Rosenfeld, and J. Z. Kolter · 2019
Closest in time.
Randaugment: Practical data augmentation with no separate search
Original
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le · 2019
Closest in time.
A kernel theory of modern data augmentation
T. Dao, A. Gu, A. Ratner, V. Smith, C. De Sa, and C. Re · 2019
Closest in time.
Equivariant multi-view networks
Original
C. Esteves, Y. Xu, C. Allen-Blanchette, and K. Daniilidis · 2019
Closest in time.
The complexity of making the gradient small in stochastic convex optimization
Original
D. Foster, A. Sekhari, O. Shamir, N. Srebro, K. Sridharan, and B. Woodworth · 2019
Closest in time.
Population based augmentation: Efficient learning of augmentation policy schedules
Original
D. Ho, E. Liang, I. Stoica, P. Abbeel, and X. Chen · 2019
Closest in time.
Augment your batch: better training with larger batches
Original
E. Hoffer, T. Ben-Nun, I. Hubara, N. Giladi, T. Hoefler, and D. Soudry · 2019
Closest in time.
A hessian based complexity measure for deep networks
Original
H. Javadi, R. Balestriero, and R. Baraniuk · 2019
Closest in time.
Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks
Original
Z. Ji and M. Telgarsky · 2019
Closest in time.
Bad global minima exist and sgd can reach them
Original
S. Liu, D. Papailiopoulos, and D. Achlioptas · 2019
Closest in time.
Improving robustness without sacrificing accuracy with patch gaussian augmentation
Original
R. G. Lopes, D. Yin, B. Poole, J. Gilmer, and E. D. Cubuk · 2019
Closest in time.
An analysis of the effect of invariance on generalization in neural networks
C. Lyle, M. Kwiatkowksa, and Y. Gal · 2019
Closest in time.
Stochastic replica voting machine prediction of stable cubic and double perovskite materials and binary alloys
T. Mazaheri, B. Sun, J. Scher-Zagier, A. Thind, D. Magee, P. Ronhovde, T. Lookman, R. Mishra, and Z. Nussinov · 2019
Closest in time.
Specaugment: A simple data augmentation method for automatic speech recognition
Original
D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le · 2019
Closest in time.
Does data augmentation lead to positive margin?
Original
S. Rajput, Z. Feng, Z. Charles, P.-L. Loh, and D. Papailiopoulos · 2019
Closest in time.
High-dimensional statistics: A non-asymptotic viewpoint , volume 48
M. J. Wainwright · 2019
Closest in time.
Unsupervised data augmentation
Original
Q. Xie, Z. Dai, E. Hovy, M.-T. Luong, and Q. V. Le · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…