2019

Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules

Ho, Daniel, Liang, Eric, Stoica, Ion et al.

Understand

A key challenge in leveraging data augmentation for neural network training is choosing an effective augmentation policy from a large search space of candidate operations.

Properly chosen augmentation policies can lead to significant generalization improvements; however, state-of-the-art approaches such as AutoAugment are computationally infeasible to run for the ordinary user.
In this paper, we introduce a new data augmentation algorithm, Population Based Augmentation (PBA), which generates nonstationary augmentation policy schedules instead of a fixed augmentation policy.
We show that PBA can match the performance of AutoAugment on CIFAR-10, CIFAR-100, and SVHN, with three orders of magnitude less overall compute.

Built on

Best practices for convolutional neural networks applied to visual document analysis
Simard, P., Steinkraus, D., and Platt, J · 2003
Earlier work this paper cites.
Natural selection fails to optimize mutation rates for long-term adaptation on rugged fitness landscapes
Clune, J., Misevic, D., Ofria, C., Lenski, R. E., Elena, S. F., and Sanjuán, R · 2008
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2009
Earlier work this paper cites.
Gaussian process bandits without regret: An experimental design approach
Original
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. W · 2009
Earlier work this paper cites.
Algorithms for hyper-parameter optimization
Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B · 2011
Earlier work this paper cites.
Sequential model-based optimization for general algorithm configuration
Hutter, F., Hoos, H. H., and Leyton-Brown, K · 2011
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Y Ng, A · 2011
Earlier work this paper cites.
Random search for hyper-parameter optimization
Bergstra, J. and Bengio, Y · 2012
Earlier work this paper cites.
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., and Schmidhuber, J · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms
Snoek, J., Larochelle, H., and Adams, R. P · 2012
Earlier work this paper cites.
Network in network
Original
Lin, M., Chen, Q., and Yan, S · 2013
Earlier work this paper cites.
Regularization of neural networks using dropconnect
Wan, L., Zeiler, M. D., Zhang, S., LeCun, Y., and Fergus, R · 2013
Earlier work this paper cites.
Fractional max-pooling
Original
Graham, B · 2014
Earlier work this paper cites.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A · 2014
Earlier work this paper cites.
Deep residual learning for image recognition
Original
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Earlier work this paper cites.

Similar

Apac: Augmented pattern classification with neural networks, 2015
Sato, I., Nishimura, H., and Yokoi, K · 2015
Cited alongside, same era.
Parallel predictive entropy search for batch global optimization of expensive objective functions
Original
Shah, A. and Ghahramani, Z · 2015
Cited alongside, same era.
Going deeper with convolutions
Original
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A · 2015
Cited alongside, same era.
Batch bayesian optimization via local penalization
González, J., Dai, Z., Hennig, P., and Lawrence, N. D · 2016
Cited alongside, same era.
Deep pyramidal residual networks
Original
Han, D., Kim, J., and Kim, J · 2016
Cited alongside, same era.
Identity mappings in deep residual networks
Original
He, K., Zhang, X., Ren, S., and Sun, J · 2016
Cited alongside, same era.
Densely connected convolutional networks
Original
Huang, G., Liu, Z., and Weinberger, K. Q · 2016
Cited alongside, same era.
Efficient hyperparameter optimization and infinitely many armed bandits
Original
Li, L., Jamieson, K. G., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A · 2016
Cited alongside, same era.
SGDR: stochastic gradient descent with restarts
Original
Loshchilov, I. and Hutter, F · 2016
Cited alongside, same era.
Bayesian optimization with robust bayesian neural networks
Springenberg, J. T., Klein, A., Falkner, S., and Hutter, F · 2016
Cited alongside, same era.
Understanding data augmentation for classification: when to warp?
Original
Wong, S. C., Gatt, A., Stamatescu, V., and McDonnell, M. D · 2016
Cited alongside, same era.
Wide residual networks
Original
Zagoruyko, S. and Komodakis, N · 2016
Cited alongside, same era.

Then

Google Vizier: A Service for Black-Box Optimization , 2017
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J. E., and Sculley, D. (eds.) · 2017
Later among the works it cites.
Population based training of neural networks
Original
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O., Green, T., Dunning, I., Simonyan, K., Fernando, C., and Kavukcuoglu, K · 2017
Later among the works it cites.
Smart augmentation - learning an optimal data augmentation strategy
Original
Lemley, J., Bazrafkan, S., and Corcoran, P · 2017
Later among the works it cites.
Ray: A distributed framework for emerging AI applications
Original
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Paul, W., Jordan, M. I., and Stoica, I · 2017
Later among the works it cites.
Improvements to context based self-supervised learning
Original
Mundhenk, T. N., Ho, D., and Chen, B. Y · 2017
Later among the works it cites.
Learning to compose domain-specific transformations for data augmentation
Original
Ratner, A. J., Ehrenberg, H. R., Hussian, Z., Dunnmon, J., and Re, C · 2017
Later among the works it cites.
Proximal policy optimization algorithms
Original
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O · 2017
Later among the works it cites.
A bayesian data augmentation approach for learning deep models
Original
Tran, T., Pham, T., Carneiro, G., Palmer, L. J., and Reid, I. D · 2017
Later among the works it cites.
Learning transferable architectures for scalable image recognition
Original
Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V · 2017
Later among the works it cites.
Gan augmentation: Augmenting training data using generative adversarial networks
Original
Bowles, C., Chen, L., Guerrero, R., Bentley, P., Gunn, R. N., Hammers, A., Dickie, D. A., del C. Valdés Hernández, M., Wardlaw, J. M., and Rueckert, D · 2018
Later among the works it cites.
Autoaugment: Learning augmentation policies from data
Original
Cubuk, E. D., Zoph, B., Mané, D., Vasudevan, V., and Le, Q. V · 2018
Later among the works it cites.
Synthetic data augmentation using gan for improved liver lesion classification
Frid-Adar, M., Klang, E., Amitai, M., Goldberger, J., and Greenspan, H · 2018
Later among the works it cites.
Data augmentation by pairing samples for images classification
Original
Inoue, H · 2018
Later among the works it cites.
An analysis of rotation matrix and colour constancy data augmentation in classifying images of animals
Okafor, E., Schomaker, L., and Wiering, M · 2018
Later among the works it cites.
Efficient neural architecture search via parameter sharing
Original
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., and Dean, J · 2018
Later among the works it cites.
Shakedrop regularization
Original
Yamada, Y., Iwamura, M., and Kise, K · 2018
Later among the works it cites.
Data augmentation with manifold exploring geometric transformations for increased performance and robustness
Original
Paschali, M., Simson, W., Guha Roy, A., Ferjad Naeem, M., Göbl, R., Wachinger, C., and Navab, N · 2019
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…