2020

AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

Erickson, Nick, Mueller, Jonas, Shirkov, Alexander et al.

Understand

We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models on an unprocessed tabular dataset such as a CSV file.

Unlike existing AutoML frameworks that primarily focus on model/hyperparameter selection, AutoGluon-Tabular succeeds by ensembling multiple models and stacking them in multiple layers.
Experiments reveal that our multi-layer combination of many models offers better use of allocated training time than seeking out the best.
A second contribution is an extensive evaluation of public and commercial AutoML platforms including TPOT, H2O, AutoWEKA, auto-sklearn, AutoGluon, and Google AutoML Tables.

Built on

Bagging predictors
Breiman, L · 1996
Earlier work this paper cites.
Experiments with a new boosting algorithm
Freund, Y., Schapire, R. E., et al · 1996
Earlier work this paper cites.
Reducing variance of committee prediction with resampling techniques
Parmanto, B., Munro, P. W., and Doyle, H. R · 1996
Earlier work this paper cites.
Stacking bagged and dagged models
Ting, K. M. and Witten, I. H · 1997
Earlier work this paper cites.
Ensemble methods in machine learning
Dietterich, T. G · 2000
Earlier work this paper cites.
Ensemble selection from libraries of models
Caruana, R., Niculescu-Mizil, A., Crew, G., and Ksikes, A · 2004
Earlier work this paper cites.
Super learner
Van der Laan, M. J., Polley, E. C., and Hubbard, A. E · 2007
Earlier work this paper cites.
The Bellkor solution to the Netflix grand prize
Koren, Y · 2009
Earlier work this paper cites.
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E · 2011
Earlier work this paper cites.
Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms
Thornton, C., Hutter, F., Hoos, H. H., and Leyton-Brown, K · 2013
Earlier work this paper cites.
Using meta-learning to initialize bayesian optimization of hyperparameters
Feurer, M., Springenberg, J. T., and Hutter, F · 2014
Earlier work this paper cites.
Hyperopt: A Python library for model selection and hyperparameter optimization
Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., and Cox, D. D · 2015
Earlier work this paper cites.
Efficient and robust automated machine learning
Feurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., and Hutter, F · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
XGBoost: A scalable tree boosting system
Chen, T. and Guestrin, C · 2016
Earlier work this paper cites.
Wide & deep learning for recommender systems
Cheng, H.-T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., Anderson, G., Corrado, G., Chai, W., Ispir, M., et al · 2016
Earlier work this paper cites.

Similar

Entity embeddings of categorical variables
Original
Guo, C. and Berkhahn, F · 2016
Cited alongside, same era.
Towards automatically-tuned neural networks
Mendoza, H., Klein, A., Feurer, M., Springenberg, J. T., and Hutter, F · 2016
Cited alongside, same era.
1st place - winner solution
Titericz, G. and Semenov, S · 2016
Cited alongside, same era.
AdaNet: Adaptive structural learning of artificial neural networks
Cortes, C., Gonzalvo, X., Kuznetsov, V., Mohri, M., and Yang, S · 2017
Cited alongside, same era.
LightGBM: A highly efficient gradient boosting decision tree
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y · 2017
Cited alongside, same era.
Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in weka
Kotthoff, L., Thornton, C., Hoos, H. H., Hutter, F., and Leyton-Brown, K · 2017
Cited alongside, same era.
Self-normalizing neural networks
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S · 2017
Cited alongside, same era.
Catboost: gradient boosting with categorical features support
Original
Dorogush, A. V., Ershov, V., and Gulin, A · 2018
Cited alongside, same era.
BOHB: Robust and efficient hyperparameter optimization at scale
Falkner, S., Klein, A., and Hutter, F · 2018
Cited alongside, same era.
Practical automated machine learning for the automl challenge 2018
Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., and Hutter, F · 2018
Cited alongside, same era.
ML-Plan: Automated machine learning via hierarchical planning
Mohr, F., Wever, M., and Hüllermeier, E · 2018
Cited alongside, same era.
TransmogrifAI
Moore, K. et al · 2018
Cited alongside, same era.

Then

GAMA: Genetic automated machine learning assistant
Gijsbers, P. and Vanschoren, J · 2019
Later among the works it cites.
An open source AutoML benchmark
Original
Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., and Vanschoren, J · 2019
Later among the works it cites.
Analysis of the automl challenge series 2015–2018
Guyon, I., Sun-Hosoya, L., Boullé, M., Escalante, H. J., Escalera, S., Liu, Z., Jajetic, D., Ray, B., Saeed, M., Sebag, M., et al · 2019
Later among the works it cites.
Automl: A survey of the state-of-the-art
Original
He, X., Zhao, K., and Chu, X · 2019
Later among the works it cites.
Auto-keras: An efficient neural architecture search system
Jin, H., Song, Q., and Hu, X · 2019
Later among the works it cites.
An end-to-end AutoML solution for tabular data at KaggleDays
Lu, Y · 2019
Later among the works it cites.
Tpot: A tree-based pipeline optimization tool for automating machine learning
Olson, R. S. and Moore, J. H · 2019
Later among the works it cites.
A Deep Dive into H2O’s AutoML, 2019
Pandey, P · 2019
Later among the works it cites.
Towards automated machine learning: Evaluation and comparison of automl approaches and tools
Original
Truong, A., Walters, A., Goodsitt, J., Hines, K., Bruss, B., and Farivar, R · 2019
Later among the works it cites.
Benchmark and survey of automated machine learning frameworks
Original
Zöller, M.-A. and Huber, M. F · 2019
Later among the works it cites.
An open source AutoML benchmark
Original
Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., and Vanschoren, J · 2019
Later among the works it cites.
An end-to-end AutoML solution for tabular data at KaggleDays
Lu, Y · 2019
Later among the works it cites.
Bringing Google AutoML to 3.5 million data scientists on Kaggle
Rishi, D · 2019
Later among the works it cites.
fastai: A layered api for deep learning
Original
Howard, J. and Gugger, S · 2020
Closest in time.
fastai: A layered api for deep learning
Original
Howard, J. and Gugger, S · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…