2020

Using a thousand optimization tasks to learn hyperparameter search strategies

Metz, Luke, Maheswaranathan, Niru, Sun, Ruoxi et al.

Understand

We present TaskSet, a dataset of tasks for use in training and evaluating optimizers.

TaskSet is unique in its size and diversity, containing over a thousand tasks ranging from image classification with fully connected or convolutional neural networks, to variational autoencoders, to non-volume preserving flows on a variety of datasets.
As an example application of such a dataset we explore meta-learning an ordered list of hyperparameters to try sequentially.
By learning this hyperparameter list from data generated using TaskSet we achieve large speedups in sample efficiency over random search.

Built on

Using learned optimizers to make models robust to input noise
Original
Metz, L., Maheswaranathan, N., Shlens, J., Sohl-Dickstein, J., and Cubuk, E. D · 1906
Earlier work this paper cites.
A method for unconstrained convex minimization problem with the rate of convergence o (1/kˆ 2)
Nesterov, Y · 1983
Earlier work this paper cites.
Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook
Schmidhuber, J · 1987
Earlier work this paper cites.
Learning a synaptic learning rule
Bengio, Y., Bengio, S., and Cloutier, J · 1990
Earlier work this paper cites.
Finding structure in time
Elman, J. L · 1990
Earlier work this paper cites.
Backpropagation through time: what it does and how to do it
Werbos, P. J · 1990
Earlier work this paper cites.
On learning how to learn learning strategies
Schmidhuber, J · 1995
Earlier work this paper cites.
Columbia object image library (coil-20)
Nene, S. A., Nayar, S. K., Murase, H., et al · 1996
Earlier work this paper cites.
Long short-term memory
Hochreiter, S. and Schmidhuber, J · 1997
Earlier work this paper cites.
No free lunch theorems for optimization
Wolpert, D. H. and Macready, W. G · 1997
Earlier work this paper cites.
The mnist database of handwritten digits
LeCun, Y · 1998
Earlier work this paper cites.
The OpenCV Library
Bradski, G · 2000
Earlier work this paper cites.
Fast subsampling performance estimates for classification algorithm selection
Petrak, J · 2000
Earlier work this paper cites.
Learning to learn using gradient descent
Hochreiter, S., Younger, A. S., and Conwell, P. R · 2001
Earlier work this paper cites.
Benchmarking optimization software with performance profiles
Dolan, E. D. and Moré, J. J · 2002
Earlier work this paper cites.
Neural architecture search with reinforcement learning
Original
Zoph, B. and Le, Q. V · 2002
Earlier work this paper cites.
Reducing the dimensionality of data with neural networks
Hinton, G. E. and Salakhutdinov, R. R · 2006
Earlier work this paper cites.
Visualizing data using t-sne
Maaten, L. v. d. and Hinton, G · 2008
Earlier work this paper cites.
Cifar-10 and cifar-100 datasets
Krizhevsky, A., Nair, V., and Hinton, G · 2009
Earlier work this paper cites.
Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y · 2010
Earlier work this paper cites.
Sun database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A · 2010
Earlier work this paper cites.
Algorithms for hyper-parameter optimization
Bergstra, J. S., Bardenet, R., Bengio, Y., and Kégl, B · 2011
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
Duchi, J., Hazan, E., and Singer, Y · 2011
Earlier work this paper cites.
Learning word vectors for sentiment analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C · 2011
Earlier work this paper cites.
Random search for hyper-parameter optimization
Bergstra, J. and Bengio, Y · 2012
Earlier work this paper cites.
Combining meta-learning and search techniques to select parameters for support vector machines
Gomes, T. A., Prudêncio, R. B., Soares, C., Rossi, A. L., and Carvalho, A · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Meta-learning for evolutionary parameter optimization of classifiers
Reif, M., Shafait, F., and Dengel, A · 2012
Earlier work this paper cites.
Practical bayesian optimization of machine learning algorithms
Snoek, J., Larochelle, H., and Adams, R. P · 2012
Earlier work this paper cites.
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude
Tieleman, T. and Hinton, G · 2012
Earlier work this paper cites.
Mujoco: A physics engine for model-based control
Todorov, E., Erez, T., and Tassa, Y · 2012
Earlier work this paper cites.
The arcade learning environment: An evaluation platform for general agents
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M · 2013
Earlier work this paper cites.
One billion word benchmark for measuring progress in statistical language modeling
Original
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., and Koehn, P · 2013
Earlier work this paper cites.
Generating sequences with recurrent neural networks
Original
Graves, A · 2013
Earlier work this paper cites.
Auto-encoding variational bayes
Original
Kingma, D. P. and Welling, M · 2013
Earlier work this paper cites.
Unit tests for stochastic optimization
Original
Schaul, T., Antonoglou, I., and Silver, D · 2013
Earlier work this paper cites.
Multi-task bayesian optimization
Swersky, K., Snoek, J., and Adams, R. P · 2013
Earlier work this paper cites.
Openml: Networked science in machine learning
Vanschoren, J., van Rijn, J. N., Bischl, B., and Torgo, L · 2013
Earlier work this paper cites.
Food-101 – mining discriminative components with random forests
Bossard, L., Guillaumin, M., and Van Gool, L · 2014
Earlier work this paper cites.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Original
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y · 2014
Earlier work this paper cites.
Using meta-learning to initialize bayesian optimization of hyperparameters
Feurer, M., Springenberg, J. T., and Hutter, F · 2014
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Kingma, D. P. and Ba, J · 2014
Earlier work this paper cites.
Freeze-thaw bayesian optimization
Original
Swersky, K., Snoek, J., and Adams, R. P · 2014
Earlier work this paper cites.
Accelerating t-sne using tree-based algorithms
Van Der Maaten, L · 2014
Earlier work this paper cites.

Similar

Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves
Domhan, T., Springenberg, J. T., and Hutter, F · 2015
Cited alongside, same era.
Efficient benchmarking of hyperparameter optimizers via surrogates
Eggensperger, K., Hutter, F., Hoos, H., and Leyton-Brown, K · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
He, K., Zhang, X., Ren, S., and Sun, J · 2015
Cited alongside, same era.
Deep learning
LeCun, Y., Bengio, Y., and Hinton, G · 2015
Cited alongside, same era.
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L · 2015
Cited alongside, same era.
Neural machine translation of rare words with subword units
Original
Sennrich, R., Haddow, B., and Birch, A · 2015
Cited alongside, same era.
Scalable bayesian optimization using deep neural networks
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., and Adams, R · 2015
Cited alongside, same era.
Learning hyperparameter optimization initializations
Wistuba, M., Schilling, N., and Schmidt-Thieme, L · 2015
Cited alongside, same era.
Sequential model-free hyperparameter tuning
Wistuba, M., Schilling, N., and Schmidt-Thieme, L · 2015
Cited alongside, same era.
Tensorflow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al · 2016
Cited alongside, same era.
Learning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., and de Freitas, N · 2016
Cited alongside, same era.
Using fast weights to attend to the recent past
Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C · 2016
Cited alongside, same era.

Then

Re-evaluating evaluation
Balduzzi, D., Tuyls, K., Perolat, J., and Graepel, T · 2018
Later among the works it cites.
JAX: composable transformations of Python+NumPy programs, 2018
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., and Wanderman-Milne, S · 2018
Later among the works it cites.
Bayesian optimization in alphago
Original
Chen, Y., Huang, A., Wang, Z., Antonoglou, I., Schrittwieser, J., Silver, D., and de Freitas, N · 2018
Later among the works it cites.
Quantifying generalization in reinforcement learning
Original
Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J · 2018
Later among the works it cites.
Bohb: Robust and efficient hyperparameter optimization at scale
Original
Falkner, S., Klein, A., and Hutter, F · 2018
Later among the works it cites.
On the selection of initialization and activation function for deep neural networks
Original
Hayou, S., Doucet, A., and Rousseau, J · 2018
Later among the works it cites.
Automatic Machine Learning: Methods, Systems, Challenges
Hutter, F., Kotthoff, L., and Vanschoren, J. (eds.) · 2018
Later among the works it cites.
Universal statistics of fisher information in deep neural networks: mean field approach
Karakida, R., Akaho, S., and Amari, S.-i · 2018
Later among the works it cites.
Parallel architecture and hyperparameter search via successive halving and classification
Original
Kumar, M., Dahl, G. E., Vasudevan, V., and Norouzi, M · 2018
Later among the works it cites.
An empirical model of large-batch training
Original
McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D · 2018
Later among the works it cites.
The natural language decathlon: Multitask learning as question answering
Original
McCann, B., Keskar, N. S., Xiong, C., and Socher, R · 2018
Later among the works it cites.
Gotta learn fast: A new benchmark for generalization in rl
Original
Nichol, A., Pfau, V., Hesse, C., Klimov, O., and Schulman, J · 2018
Later among the works it cites.
Scalable hyperparameter transfer learning
Perrone, V., Jenatton, R., Seeger, M. W., and Archambeau, C · 2018
Later among the works it cites.
Learning multiple defaults for machine learning algorithms
Original
Pfisterer, F., van Rijn, J. N., Probst, P., Müller, A., and Bischl, B · 2018
Later among the works it cites.
Critical initialisation for deep signal propagation in noisy rectifier neural networks
Pretorius, A., van Biljon, E., Kroon, S., and Kamper, H · 2018
Later among the works it cites.
Nevergrad - A gradient-free optimization platform
Rapin, J. and Teytaud, O · 2018
Later among the works it cites.
Measuring the effects of data parallelism on neural network training
Original
Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E · 2018
Later among the works it cites.
DeepMind control suite
Original
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M · 2018
Later among the works it cites.
Glue: A multi-task benchmark and analysis platform for natural language understanding
Original
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R · 2018
Later among the works it cites.
Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks
Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S., and Pennington, J · 2018
Later among the works it cites.
Open-ended learning in symmetric zero-sum games
Original
Balduzzi, D., Garnelo, M., Bachrach, Y., Czarnecki, W. M., Perolat, J., Jaderberg, M., and Graepel, T · 2019
Later among the works it cites.
A mean field theory of quantized deep networks: The quantization-depth trade-off
Original
Blumenfeld, Y., Gilboa, D., and Soudry, D · 2019
Later among the works it cites.
On empirical comparisons of optimizers for deep learning
Original
Choi, D., Shallue, C. J., Nado, Z., Lee, J., Maddison, C. J., and Dahl, G. E · 2019
Later among the works it cites.
Leveraging procedural generation to benchmark reinforcement learning, 2019
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J · 2019
Later among the works it cites.
Stochastic gradient methods with layer-wise adaptive moments for training of deep networks
Original
Ginsburg, B., Castonguay, P., Hrinchuk, O., Kuchaiev, O., Lavrukhin, V., Leary, R., Li, J., Nguyen, H., and Cohen, J. M · 2019
Later among the works it cites.
TF-Agents: A library for reinforcement learning in tensorflow
Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E., Fishman, S., Wang, K., Gonina, E., Wu, N., Kokiopoulou, E., Sbaiz, L., Smith, J., Bartók, G., Berent, J., Harris, C., Vanhoucke, V., and Brevdo, E · 2019
Later among the works it cites.
Mean-field behaviour of neural tangent kernel for deep neural networks, 2019
Hayou, S., Doucet, A., and Rousseau, J · 2019
Later among the works it cites.
Obstacle tower: A generalization challenge in vision, control, and planning
Original
Juliani, A., Khalifa, A., Berges, V.-P., Harper, J., Henry, H., Crespi, A., Togelius, J., and Lange, D · 2019
Later among the works it cites.
Tabular benchmarks for joint architecture and hyperparameter optimization
Original
Klein, A. and Hutter, F · 2019
Later among the works it cites.
On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training
Li, P. and Nguyen, P.-M · 2019
Later among the works it cites.
On the variance of the adaptive learning rate and beyond
Original
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J · 2019
Later among the works it cites.
DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning
Olsen, A., Konovalov, D. A., Philippa, B., Ridd, P., Wood, J. C., Johns, J., Banks, W., Girgenti, B., Kenny, O., Whinney, J., Calvert, B., Rahimi Azghadi, M., and White, R. D · 2019
Later among the works it cites.
Solving rubik’s cube with a robot hand
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M., Powell, G., Ribas, R., Schneider, J., Tezak, N., Tworek, J., Welinder, P., Weng, L., Yuan, Q., Zaremba, W., and Zhang, L · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S · 2019
Later among the works it cites.
Learning search spaces for bayesian optimization: Another view of hyperparameter transfer learning
Perrone, V. and Shen, H · 2019
Later among the works it cites.
Deepobs: A deep learning optimizer benchmark suite
Schneider, F., Balles, L., and Hennig, P · 2019
Later among the works it cites.
On the tunability of optimizers in deep learning
Original
Sivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., and Fleuret, F · 2019
Later among the works it cites.
Energy and policy considerations for deep learning in nlp
Original
Strubell, E., Ganesh, A., and McCallum, A · 2019
Later among the works it cites.
tensorflow tpu resnet50, Oct 2019
Tensorflow · 2019
Later among the works it cites.
Meta-dataset: A dataset of datasets for learning to learn from few examples
Original
Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Xu, K., Goroshin, R., Gelada, C., Swersky, K., Manzagol, P.-A., and Larochelle, H · 2019
Later among the works it cites.
Superglue: A stickier benchmark for general-purpose language understanding systems
Original
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R · 2019
Later among the works it cites.
NAS-Bench-101: Towards Reproducible Neural Architecture Search
Original
Ying, C., Klein, A., Real, E., Christiansen, E., Murphy, K., and Hutter, F · 2019
Later among the works it cites.
The visual task adaptation benchmark
Original
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al · 2019
Later among the works it cites.
Flax: A neural network library for jax designed for flexibility, 2020
Flax Developers · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…