2020

FMix: Enhancing Mixed Sample Data Augmentation

Harris, Ethan, Marcu, Antonia, Painter, Matthew et al.

Understand

Mixed Sample Data Augmentation (MSDA) has received increasing attention in recent years, with many successful variants such as MixUp and CutMix.

By studying the mutual information between the function learned by a VAE on the original data and on the augmented data we show that MixUp distorts learned functions in a way that CutMix does not.
We further demonstrate this by showing that MixUp acts as a form of adversarial training, increasing robustness to attacks such as Deep Fool and Uniform Noise which produce examples similar to those generated by MixUp.
We argue that this distortion prevents models from learning about sample specific features in the data, aiding generalisation performance.

Built on

Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 1912–1920
1920
Earlier work this paper cites.
Y. LeCun, Y. Bengio et al. , “Convolutional networks for images, speech, and time series,” The handbook of brain theory and neural networks , vol. 3361, no. 10, p. 1995, 1995
1995
Earlier work this paper cites.
S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation , vol. 9, no. 8, pp. 1735–1780, 1997
1997
Earlier work this paper cites.
V. Vapnik, The nature of statistical learning theory . Springer science & business media, 1999
1999
Earlier work this paper cites.
O. Chapelle, J. Weston, L. Bottou, and V. Vapnik, “Vicinal risk minimization,” in Advances in neural information processing systems , 2001, pp. 416–422
2001
Earlier work this paper cites.
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” Journal of artificial intelligence research , vol. 16, pp. 321–357, 2002
2002
Earlier work this paper cites.
A. Krizhevsky et al. , “Learning multiple layers of features from tiny images,” 2009
2009
Earlier work this paper cites.
A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1 . Association for Computational Linguistics, 2011, pp. 142–150
2011
Earlier work this paper cites.
D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114 , 2013
Original
2013
Earlier work this paper cites.
I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572 , 2014
Original
2014
Earlier work this paper cites.
N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in 2015 IEEE Information Theory Workshop (ITW) . IEEE, 2015, pp. 1–5
2015
Earlier work this paper cites.
Stanford, “Tiny imagenet visual recognition challenge,” 2015. [Online]. Available: https://tiny-imagenet.herokuapp.com/
2015
Earlier work this paper cites.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” International Journal of Computer Vision (IJCV) , vol. 115, no. 3, pp. 211–252, 2015
2015
Earlier work this paper cites.
X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” in Advances in neural information processing systems , 2015, pp. 649–657
2015
Earlier work this paper cites.
S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 2574–2582
2016
Earlier work this paper cites.
M. D. Hoffman and M. J. Johnson, “Elbo surgery: yet another way to carve up the variational evidence lower bound,” in Workshop in Advances in Approximate Bayesian Inference, NIPS , vol. 1, 2016, p. 2
2016
Earlier work this paper cites.
K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in European conference on computer vision . Springer, 2016, pp. 630–645
2016
Earlier work this paper cites.
S. Zagoruyko and N. Komodakis, “Wide residual networks,” arXiv preprint arXiv:1605.07146 , 2016
Original
2016
Earlier work this paper cites.
A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov, “Bag of tricks for efficient text classification,” arXiv preprint arXiv:1607.01759 , 2016
Original
2016
Earlier work this paper cites.
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” arXiv preprint arXiv:1710.09412 , 2017
Original
2017
Earlier work this paper cites.

Similar

Y. Tokozume, Y. Ushiku, and T. Harada, “Learning from between-class examples for deep sound recognition,” arXiv preprint arXiv:1711.10282 , 2017
Original
2017
Cited alongside, same era.
A. Kolchinsky and B. D. Tracey, “Estimating mixture entropy with pairwise distances,” Entropy , vol. 19, no. 7, p. 361, 2017
2017
Cited alongside, same era.
R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 618–626
2017
Cited alongside, same era.
J. Rauber, W. Brendel, and M. Bethge, “Foolbox: A python toolbox to benchmark the robustness of machine learning models,” in Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning , 2017. [Online]. Available: http://arxiv.org/abs/1707.04131
Original
2017
Cited alongside, same era.
H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747 , 2017
Original
2017
Cited alongside, same era.
Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” arXiv preprint arXiv:1708.04896 , 2017
Original
2017
Cited alongside, same era.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 4700–4708
2017
Cited alongside, same era.
D. Han, J. Kim, and J. Kim, “Deep pyramidal residual networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 5927–5935
2017
Cited alongside, same era.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 1492–1500
2017
Cited alongside, same era.
——, “Between-class learning for image classification,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 5486–5494
2018
Cited alongside, same era.
H. Inoue, “Data augmentation by pairing samples for images classification,” arXiv preprint arXiv:1801.02929 , 2018
Original
2018
Cited alongside, same era.
D. Liang, F. Yang, T. Zhang, and P. Yang, “Understanding mixup training methods,” IEEE Access , vol. 6, pp. 58 774–58 783, 2018
2018
Cited alongside, same era.

Then

N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin, “On evaluating adversarial robustness,” arXiv preprint arXiv:1902.06705 , 2019
Original
2019
Later among the works it cites.
H. Guo, Y. Mao, and R. Zhang, “Mixup as locally linear out-of-manifold regularization,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, 2019, pp. 3714–3722
2019
Later among the works it cites.
Z. He, L. Xie, X. Chen, Y. Zhang, Y. Wang, and Q. Tian, “Data augmentation revisited: Rethinking the distribution gap between clean and augmented data,” arXiv preprint arXiv:1909.09148 , 2019
Original
2019
Later among the works it cites.
V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in Proceedings of the 36th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. Long Beach, California, USA: PMLR, 09–15 Jun 2019, pp. 6438–6447
2019
Later among the works it cites.
S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim, “Fast autoaugment,” in Advances in Neural Information Processing Systems (NeurIPS) , 2019
2019
Later among the works it cites.
E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation strategies from data,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2019, pp. 113–123
2019
Later among the works it cites.
D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple data processing method to improve robustness and uncertainty,” arXiv preprint arXiv:1912.02781 , 2019
Original
2019
Later among the works it cites.
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural adversarial examples,” arXiv preprint arXiv:1907.07174 , 2019
Original
2019
Later among the works it cites.
T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natural language processing,” ArXiv , vol. abs/1910.03771, 2019
Original
2019
Later among the works it cites.
E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” arXiv preprint arXiv:1909.13719 , 2019
Original
2019
Later among the works it cites.
J.-H. Kim, W. Choo, and H. O. Song, “Puzzle mix: Exploiting saliency and local statistics for optimal mixup,” arXiv preprint arXiv:2009.06962 , 2020
Original
2020
Closest in time.
D. Walawalkar, Z. Shen, Z. Liu, and M. Savvides, “Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 3642–3646
2020
Closest in time.
A. Uddin, M. Monira, W. Shin, T. Chung, S.-H. Bae et al. , “Saliencymix: A saliency guided data augmentation strategy for better regularization,” arXiv preprint arXiv:2006.01791 , 2020
Original
2020
Closest in time.
G. Perrault-Archambault, Y. Mao, H. Guo, and R. Zhang, “Mixup as directional adversarial training,” 2020. [Online]. Available: https://openreview.net/forum?id=SkgjKR4YwH
2020
Closest in time.
R. Gontijo-Lopes, S. J. Smullin, E. D. Cubuk, and E. Dyer, “Affinity and diversity: Quantifying mechanisms of data augmentation,” arXiv preprint arXiv:2002.08973 , 2020
Original
2020
Closest in time.
L. Carratino, M. Cissé, R. Jenatton, and J.-P. Vert, “On mixup regularization,” arXiv preprint arXiv:2006.06049 , 2020
Original
2020
Closest in time.
L. Zhang, Z. Deng, K. Kawaguchi, A. Ghorbani, and J. Zou, “How does mixup help with robustness and generalization?” arXiv preprint arXiv:2010.04819 , 2020
Original
2020
Closest in time.
J. Rauber, R. Zimmermann, M. Bethge, and W. Brendel, “Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax,” Journal of Open Source Software , vol. 5, no. 53, p. 2607, 2020. [Online]. Available: https://doi.org/10.21105/joss.02607
2020
Closest in time.
S. Alam, T. Reasat, A. S. Sushmit, S. M. Siddiquee, F. Rahman, M. Hasan, and A. I. Humayun, “Multi-label classification of common bengali handwritten graphemes: Dataset and challenge,” 2020
2020
Closest in time.
P. Singer and D. Gordeev, “Bengali.ai handwritten grapheme classification competition: Second place solution,” 2020. [Online]. Available: https://www.kaggle.com/c/bengaliai-cv19/discussion/135966
2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…