2018

Automated Machine Learning: From Principles to Practices

Shen, Zhenqian, Zhang, Yongqi, Wei, Lanning et al.

Understand

Machine learning (ML) methods have been developing rapidly, but configuring and selecting proper methods to achieve a desired performance is increasingly difficult and tedious.

To address this challenge, automated machine learning (AutoML) has emerged, which aims to generate satisfactory ML configurations for given tasks in a data-driven way.
In this paper, we provide a comprehensive survey on this topic.
We begin with the formal definition of AutoML and then introduce its principles, including the bi-level learning objective, the learning strategy, and the theoretical interpretation.

Built on

Language models are few-shot learners
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al · 1901
Earlier work this paper cites.
Genetic algorithms as a tool for feature selection in machine learning. In International Conference on Tools with Artificial Intelligence . 200–203
H. Vafaie and K. De Jong. 1992 · 1992
Earlier work this paper cites.
Support-vector networks
C. Cortes and V. Vapnik. 1995 · 1995
Earlier work this paper cites.
Machine Learning
T. Mitchell. 1997 · 1997
Earlier work this paper cites.
Support vector clustering
A. Ben-Hur, D. Horn, H. T. Siegelmann, and V. Vapnik. 2001 · 2001
Earlier work this paper cites.
Foundations of bilevel programming
S. Dempe. 2002 · 2002
Earlier work this paper cites.
Genetic programming with a genetic algorithm for feature construction and selection
M. Smith and L. Bull. 2005 · 2005
Earlier work this paper cites.
Getting the most out of ensemble selection. In International Conference on Data Mining . 828–833
R. Caruana, A. Munson, and A. Niculescu-Mizil. 2006 · 2006
Earlier work this paper cites.
Compressed sensing
D. L. Donoho. 2006 · 2006
Earlier work this paper cites.
The tradeoffs of large scale learning. In Advances in Neural Information Processing Systems . 161–168
L. Bottou and O. Bousquet. 2008 · 2008
Earlier work this paper cites.
Introduction to derivative-free optimization
A. R. Conn, k. Scheinberg, and L. N. Vicente. 2009 · 2009
Earlier work this paper cites.
A Survey on Transfer Learning
S. Pan and Q. Yang. 2009 · 2009
Earlier work this paper cites.
Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning . 807–814
V. Nair and G. Hinton. 2010 · 2010
Earlier work this paper cites.
Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems . 2546–2554
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl. 2011 · 2011
Earlier work this paper cites.
Adaptive subgradient methods for online learning and stochastic optimization
J. Duchi, E. Hazan, and Y. Singer. 2011 · 2011
Earlier work this paper cites.
Scikit-learn: Machine Learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011 · 2011
Earlier work this paper cites.
Random search for hyper-parameter optimization
J. Bergstra and Y. Bengio. 2012 · 2012
Earlier work this paper cites.
Imagenet: classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems . 1097–1105
A. Krizhevsky, I. Sutskever, and G. Hinton. 2012 · 2012
Earlier work this paper cites.
Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning . 1238–1246
Z. Karnin, T. Koren, and O. Somekh. 2013 · 2013
Earlier work this paper cites.
Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 847–855
C. Thornton, F. Hutter, H. Hoos, and K. Leyton-Brown. 2013 · 2013
Earlier work this paper cites.
A survey on feature selection methods
G. Chandrashekar and F. Sahin. 2014 · 2014
Earlier work this paper cites.
Raiders of the lost architecture: Kernels for Bayesian optimization in conditional parameter spaces
Original
K. Swersky, D. Duvenaud, J. Snoek, F. Hutter, and M. Osborne. 2014 · 2014
Earlier work this paper cites.
Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In International Joint Conference on Artificial Intelligence
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. 2015 · 2015
Earlier work this paper cites.
Efficient Benchmarking of Hyperparameter Optimizers via Surrogates. In AAAI Conference on Artificial Intelligence
K. Eggensperger, F. Hutter, H. Hoos, and K. Leyton-Brown. 2015 · 2015
Earlier work this paper cites.
Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems . 2962–2970
M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and F. Hutter. 2015 · 2015
Earlier work this paper cites.
Unsupervised domain adaptation by backpropagation. In International Conference on Machine Learning . 1180–1189
Y. Ganin and V. Lempitsky. 2015 · 2015
Earlier work this paper cites.
Fast r-cnn. In International Conference on Computer Vision . 1440–1448
R. Girshick. 2015 · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
D. P. Kingma and J. Ba. 2015 · 2015
Earlier work this paper cites.
Bi-level stochastic gradient for large scale support vector machine. In Neurocomputing . 300–308
C. Nicolas and W. Wang. 2015 · 2015
Earlier work this paper cites.
Faster r-cnn: Towards real-time object detection with region proposal networks
S. Ren, K. He, R. Girshick, and J. Sun. 2015 · 2015
Earlier work this paper cites.
Optimizing deep learning hyper-parameters through an evolutionary algorithm. In Workshop on Machine Learning in High-performance Computing Environments
S. Young, D. Rose, T. Karnowski, S. Lim, and R. Patton. 2015 · 2015
Earlier work this paper cites.
Deep learning
I. Goodfellow, Y. Bengio, and A. Courville. 2016 · 2016
Earlier work this paper cites.
Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition . 770–778
K. He, X. Zhang, S. Ren, and J. Sun. 2016 · 2016
Earlier work this paper cites.
Non-stochastic best arm identification and hyperparameter optimization. In Artificial Intelligence and Statistics . 240–248
K. Jamieson and A. Talwalkar. 2016 · 2016
Earlier work this paper cites.
Learning curve prediction with Bayesian neural networks. In International Conference on Learning Representations
A. Klein, S. Falkner, J. T. Springenberg, and F. Hutter. 2016 · 2016
Earlier work this paper cites.
SGDR: Stochastic Gradient Descent with Warm Restarts. In International Conference on Learning Representations
I. Loshchilov and F. Hutter. 2016 · 2016
Earlier work this paper cites.
Evaluation of a tree-based pipeline optimization tool for automating data science. In Genetic and Evolutionary Computation Conference . 485–492
R. Olson, N. Bartley, R. Urbanowicz, and J. Moore. 2016 · 2016
Earlier work this paper cites.
TPOT: A tree-based pipeline optimization tool for automating machine learning. In Workshop on Automatic Machine Learning . 66–74
R. Olson and J. Moore. 2016 · 2016
Earlier work this paper cites.
Hyperparameter optimization with approximate gradient. In International Conference on Machine Learning . 737–746
F. Pedregosa. 2016 · 2016
Earlier work this paper cites.
You only look once: Unified, real-time object detection. In Conference on Computer Vision and Pattern Recognition . 779–788
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi. 2016 · 2016
Earlier work this paper cites.
Convolutional neural fabrics
S. Saxena and J. Verbeek. 2016 · 2016
Earlier work this paper cites.
Accelerating neural architecture search using performance prediction
Original
B. Baker, O. Gupta, R. Raskar, and N. Naik. 2017b · 2017
Earlier work this paper cites.
A downsampled variant of imagenet as an alternative to the cifar datasets
Original
P. Chrabaszcz, I. Loshchilov, and F. Hutter. 2017 · 2017
Earlier work this paper cites.
AdaNet: Adaptive Structural Learning of Artificial Neural Networks. In International Conference on Machine Learning . 874–883
C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, and S. Yang. 2017 · 2017
Earlier work this paper cites.
Neural architecture search: A survey
T. Elsken, J. Metzen, and F. Hutter. 2019 · 2017
Earlier work this paper cites.
Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning . 1126–1135
C. Finn, P. Abbeel, and S. Levine. 2017 · 2017
Earlier work this paper cites.
How machine learning could help to improve climate forecasts
N. Jones. 2017 · 2017
Earlier work this paper cites.
Hyperband: A novel bandit-based approach to hyperparameter optimization
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. 2017 · 2017
Earlier work this paper cites.
SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. 2017 · 2017
Earlier work this paper cites.
Large-scale evolution of image classifiers. In International Conference on Machine Learning
E. Real, S. Moore, A. Selle, S. Saxena, Y. Suematsu, J. Tan, Q. Le, and A. Kurakin. 2017 · 2017
Earlier work this paper cites.
Attention is all you need
A. Vaswani, N. Shazeer, N. Parmar, et al · 2017
Earlier work this paper cites.
Neural architecture search with reinforcement learning. In International Conference on Learning Representations
B. Zoph and Q. Le. 2017 · 2017
Earlier work this paper cites.
Learning Transferable Architectures for Scalable Image Recognition. In Conference on Computer Vision and Pattern Recognition
B. Zoph, V. Vasudevan, J. Shlens, and Q. Le. 2017 · 2017
Earlier work this paper cites.
Understanding and Simplifying One-Shot Architecture Search. In International Conference on Machine Learning . 549–558
G. Bender, P. J. Kindermans, B. Zoph, V. Vasudevan, and Q. Le. 2018 · 2018
Earlier work this paper cites.
BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning . 1437–1446
S. Falkner, A. Klein, and F. Hutter. 2018 · 2018
Earlier work this paper cites.

Similar

A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation. In International Conference on Learning Representations
A. Gotmare, N. S. Keskar, C. Xiong, and R. Socher. 2018 · 2018
Cited alongside, same era.
Derivative free optimization via repeated classification
T. Hashimoto, S. Yadlowsky, and J. Duchi. 2018 · 2018
Cited alongside, same era.
Hyperparameter optimization: a spectral approach. In International Conference on Learning Representations
E. Hazan, A. Klivans, and Y. Yuan. 2018 · 2018
Cited alongside, same era.
AMC: AutoML for Model Compression and Acceleration on Mobile Devices. In European Conference on Computer Vision . 784–800
Y. He, J. Lin, Z. Liu, H. Wang, L. Li, and S. Han. 2018 · 2018
Cited alongside, same era.
Experienced Optimization with Reusable Directional Model for Hyper-Parameter Search.. In International Joint Conference on Artificial Intelligence . 2276–2282
Y. Hu, Y. Yu, and Z. Zhou. 2018 · 2018
Cited alongside, same era.
Neural architecture search with bayesian optimisation and optimal transport
K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing. 2018 · 2018
Cited alongside, same era.
DARTS: Differentiable Architecture Search
H. Liu, K. Simonyan, and Y. Yang. 2018a · 2018
Cited alongside, same era.
Neural architecture optimization
R. Luo, F. Tian, T. Qin, E. Chen, and T. Liu. 2018 · 2018
Cited alongside, same era.
Foundations of machine learning
M. Mohri, A. Rostamizadeh, and A. Talwalkar. 2018 · 2018
Cited alongside, same era.
Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning . 4095–4104
H. Pham, M. Guan, B. Zoph, Q. Le, and J. Dean. 2018 · 2018
Cited alongside, same era.
Automating drug discovery
G. Schneider. 2018 · 2018
Cited alongside, same era.
Deep learning in the wild. In International Workshop on Challenges in Artificial Intelligence and Machine Learning for Internet of Things . 17–38
T. Stadelmann, M. Amirian, I. Arabaci, et al · 2018
Cited alongside, same era.

Then

Understanding and Robustifying Differentiable Architecture Search. In International Conference on Learning Representations
A. Zela, T. Elsken, T. Saikia, Y. Marrakchi, T. Brox, and F. Hutter. 2020 · 2020
Closest in time.
You only search once: Single shot neural architecture search via direct sparse optimization
X. Zhang, Z. Huang, N. Wang, S. Xiang, and C. Pan. 2020 · 2020
Closest in time.
Bootstrapped meta-learning
Original
S. Flennerhag, Y. Schroecker, T. Zahavy, H. van Hasselt, D. Silver, and S. Singh. 2021 · 2021
Closest in time.
AutoML: A survey of the state-of-the-art
X. He, K. Zhao, and X. Chu. 2021 · 2021
Closest in time.
Meta-learning in neural networks: A survey
T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. 2021 · 2021
Closest in time.
RankNAS: Efficient Neural Architecture Search by Pairwise Ranking. In Conference on Empirical Methods in Natural Language Processing . 2469–2480
C. Hu, C. Wang, X. Ma, X. Meng, Y. Li, T. Xiao, J. Zhu, and C. Li. 2021 · 2021
Closest in time.
Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Annual Meeting of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing . 4582–4597
X. Li and P. Liang. 2021 · 2021
Closest in time.
Zen-nas: A zero-shot nas for high-performance image recognition. In International Conference on Computer Vision . 347–356
M. Lin, P. Wang, Z. Sun, H. Chen, X. Sun, Q. Qian, H. Li, and R. Jin. 2021 · 2021
Closest in time.
Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond
R. Liu, J. Gao, J. Zhang, D. Meng, and Z. Lin. 2021b · 2021
Closest in time.
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. In Annual Meeting of the Association for Computational Linguistics
R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. 2021 · 2021
Closest in time.
Neural architecture search without training. In International Conference on Machine Learning . 7588–7598
J. Mellor, J. Turner, A. Storkey, and E. J. Crowley. 2021 · 2021
Closest in time.
Accelerating neural architecture search via proxy data
B. Na, J. Mok, H. Choe, and S. Yoon. 2021 · 2021
Closest in time.
Primer: Searching for efficient transformers for language modeling
D. So, W. Mańke, H. Liu, Z. Dai, N. Shazeer, and Q. Le. 2021 · 2021
Closest in time.
Neural architecture search as sparse supernet. In AAAI Conference on Artificial Intelligence . 10379–10387
Y. Wu, A. Liu, Z. Huang, S. Zhang, and L. Van Gool. 2021 · 2021
Closest in time.
Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search. In Conference on Computer Vision and Pattern Recognition . 13723–13732
K. Yu, R. Ranftl, and M. Salzmann. 2021 · 2021
Closest in time.
Few-shot neural architecture search. In International Conference on Machine Learning . 12707–12718
Y. Zhao, L. Wang, Y. Tian, R. Fonseca, and T. Guo. 2021 · 2021
Closest in time.
Machine learning
Z. Zhou. 2021 · 2021
Closest in time.
Characterization of AI Model Configurations for Model Reuse. In European Conference on Computer Vision . 454–469
P. Bajcsy, M. Majurski, T. E. Cleveland IV, M. Carrasco, and W. Keyrouz. 2022 · 2022
Closest in time.
On the Opportunities and Risks of Foundation Models
R. Bommasani, D. A. Hudson, E. Adeli, et al · 2022
Closest in time.
Benchmarking AutoML for regression tasks on small tabular data in materials design
F. Conra, M. Mälzer, M. Schwarzenberger, H. Wiemer, and S. Ihlenfeldt. 2022 · 2022
Closest in time.
Auto-sklearn 2.0: Hands-free automl via meta-learning
M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. 2022 · 2022
Closest in time.
LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations
E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. 2022 · 2022
Closest in time.
Machine learning in the search for new fundamental physics
G. Karagiorgi, G. Kasieczka, S. Kravitz, B. Nachman, and D. Shih. 2022 · 2022
Closest in time.
Automata: Gradient based data subset selection for compute-efficient hyper-parameter tuning
K. Killamsetty, G. S. Abhishek, A. Lnu, G. Ramakrishnan, A. Evfimievski, L. Popa, and R. Iyer. 2022 · 2022
Closest in time.
Nas-bench-nlp: neural architecture search benchmark for natural language processing
N. Klyuchnikov, I. Trofimov, E. Artemova, M. Salnikov, M. Fedorov, A. Filippov, and E. Burnaev. 2022 · 2022
Closest in time.
A fast post-training pruning framework for transformers
W. Kwon, S. Kim, M. Mahoney, J. Hassoun, K. Keutzer, and A. Gholami. 2022 · 2022
Closest in time.
P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Annual Meeting of the Association for Computational Linguistics . 61–68
X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang. 2022 · 2022
Closest in time.
Training language models to follow instructions with human feedback
L. Ouyang, J. Wu, X. Jiang, et al · 2022
Closest in time.
Zero-Shot AutoML with Pretrained Models. In International Conference on Machine Learning . 17138–17155
E. Öztürk, F. Ferreira, H. Jomaa, L. Schmidt-Thieme, J. Grabocka, and F. Hutter. 2022 · 2022
Closest in time.
Speeding up NAS with adaptive subset selection. In Conference on Automated Machine Learning
V. Prasad, C. White, P. Jain, S. Nayak, R. Iyer, and G. Ramakrishnan. 2022 · 2022
Closest in time.
High-resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition . 10684–10695
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. 2022 · 2022
Closest in time.
On the convergence theory for hessian-free bilevel algorithms
D. Sow, K. Ji, and Y. Liang. 2022 · 2022
Closest in time.
Just Add Data: automated predictive modeling for knowledge discovery and feature selection
I. Tsamardinos, P. Charonyktakis, G. Papoutsoglou, et al · 2022
Closest in time.
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. In Conference on Empirical Methods in Natural Language Processing
Y. Wang, S. Agarwal, S. Mukherjee, X. Liu, J. Gao, A. Awadallah, and J. Gao. 2022 · 2022
Closest in time.
Towards effective and generalizable fine-tuning for pre-trained molecular graph models
J. Xia, J. Zheng, C. Tan, G. Wang, and S. Li. 2022 · 2022
Closest in time.
Searching a high performance feature extractor for text recognition network
H. Zhang, Q. Yao, J. Kwok, and X. Bai. 2022 · 2022
Closest in time.
GradSign: Model Performance Inference with Theoretical Insights. In International Conference on Learning Representations
Z. Zhang and Z. Jia. 2022 · 2022
Closest in time.
Mixpath: A unified approach for one-shot neural architecture search. In International Conference on Computer Vision . 5972–5981
X. Chu, S. Lu, X. Li, and B. Zhang. 2023 · 2023
Closest in time.
Parameter-efficient fine-tuning of large-scale pre-trained language models
N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C. Chan, W. Chen, et al · 2023
Closest in time.
AI computing reaches for the edge
S. S. Iyer and V. Roychowdhury. 2023 · 2023
Closest in time.
Zero-Shot Neural Architecture Search: Challenges, Solutions, and Opportunities
Original
G. Li, D. Hoang, K. Bhardwaj, M. Lin, Z. Wang, and R. Marculescu. 2023a · 2023
Closest in time.
VolcanoML: speeding up end-to-end AutoML via scalable search space decomposition
Y. Li, Y. Shen, W. Zhang, C. Zhang, and B. Cui. 2023b · 2023
Closest in time.
Towards graph foundation models: A survey and beyond
Original
J. Liu, C. Yang, Z. Lu, J. Chen, Y. Li, M. Zhang, T. Bai, Y. Fang, L. Sun, P. Yu, et al · 2023
Closest in time.
Hierarchical automated machine learning (AutoML) for advanced unconventional reservoir characterization
Y. Mubarak and A. Koeshidayatullah. 2023 · 2023
Closest in time.
Unleashing the Power of Gradient Signal-to-Noise Ratio for Zero-Shot NAS. In International Conference on Computer Vision . 5763–5773
Z. Sun, Y. Sun, L. Yang, S. Lu, J. Mei, W. Zhao, and Y. Hu. 2023 · 2023
Closest in time.
AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks
Original
A. Tornede, D. Deng, T. Eimer, et al · 2023
Closest in time.
Llama: Open and efficient foundation language models
Original
H. Touvron, T. Lavril, G. Izacard, et al · 2023
Closest in time.
An improved hyperparameter optimization framework for AutoML systems using evolutionary algorithms
A. Vincent and P. Jidesh. 2023 · 2023
Closest in time.
Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inference
C. Wang, X. Liu, and A. Awadallah. 2023b · 2023
Closest in time.
Neural architecture search: Insights from 1000 papers
Original
C. White, M. Safari, R. Sukthanker, B. Ru, T. Elsken, A. Zela, D. Dey, and F. Hutter. 2023 · 2023
Closest in time.
A comprehensive survey on pretrained foundation models: A history from bert to chatgpt
Original
C. Zhou, Q. Li, C. Li, et al · 2023
Closest in time.
AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning
Original
H. Zhou, X. Wan, I. Vulić, and A. Korhonen. 2023b · 2023
Closest in time.
Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How. In International Conference on Learning Representations
S. P. Arango, F. Ferreira, A. Kadra, F. Hutter, and J. Grabocka. 2024 · 2024
Closest in time.
Automated Machine Learning for Positive-Unlabelled Learning
Original
J. Saunders and A. Freitas. 2024 · 2024
Closest in time.
Doing more with less: characterizing dataset downsampling for automl
F. Zogaj, J. Cambronero, M. Rinard, and J. Cito. 2021 · 2072
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…