2017

OpenML Benchmarking Suites

Bischl, Bernd, Casalicchio, Giuseppe, Feurer, Matthias et al.

Understand

Machine learning research depends on objectively interpretable, comparable, and reproducible algorithm benchmarks.

We advocate the use of curated, comprehensive suites of machine learning tasks to standardize the setup, execution, and reporting of benchmarks.
We enable this through software tools that help to create and leverage these benchmarking suites.
These are seamlessly integrated into the OpenML platform, and accessible through interfaces in Python, Java, and R.

Built on

Generalizing from case studies: A case study
D. W. Aha · 1992
Earlier work this paper cites.
Smote: Synthetic minority over-sampling technique
N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer · 2002
Earlier work this paper cites.
On the need for time series data mining benchmarks: A survey and empirical demonstration
E. Keogh and S. Kasetty · 2003
Earlier work this paper cites.
Auto-sklearn 2.0: Hands-free automl via meta-learning
Original
M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter · 2007
Earlier work this paper cites.
Empiricism is not a matter of faith
T. Pedersen · 2008
Earlier work this paper cites.
The WEKA data mining software: An update
M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten · 2009
Earlier work this paper cites.
Keel datamining software tool: Data set repository, integration of algorithms and experimental analysis framework
J. Alcala, A. Fernandez, J. Luengo, J. Derrac, S. Garcia, L. Sanchez, and F. Herrera · 2010
Earlier work this paper cites.
LIBSVM: A library for support vector machines
C. C. Chang and C. J. Lin · 2011
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay · 2011
Earlier work this paper cites.
Mulan: A java library for multi-label learning
G. Tsoumakas, E. Spyromitros-Xioufis, J. Vilcek, and I. Vlahavas · 2011
Earlier work this paper cites.
Auto-weka: Combined selection and hyperparameter optimization of classification algorithms
C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown · 2013
Earlier work this paper cites.
OpenML: Networked science in machine learning
J. Vanschoren, J. N. van Rijn, B. Bischl, and L. Torgo · 2013
Earlier work this paper cites.
Skdata: data sets and algorithm evaluation protocols in python
J. Bergstra, N. Pinto, and D. Cox · 2015
Earlier work this paper cites.
The UCR time series classification archive, July 2015
Y. Chen, E. Keogh, B. Hu, N. Begum, A. Bagnall, A. Mueen, and G. Batista · 2015
Earlier work this paper cites.
OpenAI Gym
Original
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba · 2016
Earlier work this paper cites.
Massively collaborative machine learning
J. N. van Rijn · 2016
Earlier work this paper cites.
OpenML: An R package to connect to the machine learning platform OpenML
G. Casalicchio, J. Bossek, M. Lang, D. Kirchhoff, P. Kerschke, B. Hofner, H. Seibold, J. Vanschoren, and B. Bischl · 2017
Earlier work this paper cites.

Similar

UCI machine learning repository, 2017
D. Dheeru and E. K. Taniskidou · 2017
Cited alongside, same era.
PMLB: A large benchmark suite for machine learning evaluation and comparison
R. S. Olson, W. La Cava, P. Orzechowski, R. J. Urbanowicz, and J. H. Moore · 2017
Cited alongside, same era.
Alphad3m: Machine learning pipeline synthesis
I. Drori, Y. Krishnamurthy, R. Rampin, R. Lourenço, J. One, K. Cho, C. Silva, and J. Freire · 2018
Cited alongside, same era.
Datasheets for datasets
Original
T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. Daumé III, and K. Crawford · 2018
Cited alongside, same era.
P4ml: A phased performance-based pipeline planner for automated machine learning
Y. Gil, K.-T. Yao, V. Ratnakar, D. Garijo, G. Ver Steeg, P. Szekely, R. Brekelmans, M. Kejriwal, F. Luo, and I.-H. Huang · 2018
Cited alongside, same era.
Missing data hinder replication of artificial intelligence studies
M. Hutson · 2018
Cited alongside, same era.
Ml-plan: Automated machine learning via hierarchical planning
F. Mohr, M. Wever, and E. Hüllermeier · 2018
Cited alongside, same era.
Winner’s curse? on pace, progress, and empirical rigor
D. Sculley, J. Snoek, A. Wiltschko, and A. Rahimi · 2018
Cited alongside, same era.
Hyperparameter importance across datasets
J. N. van Rijn and F. Hutter · 2018
Cited alongside, same era.
An open source automl benchmark
P. Gijsbers, E. LeDell, J. Thomas, S. Poirier, B. Bischl, and J. Vanschoren · 2019
Cited alongside, same era.
Analysis of the automl challenge series 2015–2018
I. Guyon, L. Sun-Hosoya, M. Boullé, H. J. Escalante, S. Escalera, Z. Liu, D. Jajetic, B. Ray, M. Saeed, M. Sebag, A. Statnikov, W.-W. Tu, and E. Viegas · 2019
Cited alongside, same era.
mlr3: A modern object-oriented machine learning framework in r
M. Lang, M. Binder, J. Richter, P. Schratz, F. Pfisterer, S. Coors, Q. Au, G. Casalicchio, L. Kotthoff, and B. Bischl · 2019
Cited alongside, same era.

Then

Distribution-based invariant deep networks for learning meta-features
Original
G. De Bie, H. Rakotoarison, G. Peyré, and M. Sebag · 2020
Closest in time.
Autogluon-tabular: Robust and accurate automl for structured data
Original
N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Larroy, M. Li, and A. Smola · 2020
Closest in time.
Family and prejudice: A behavioural taxonomy of machine learning techniques
R. Fabra-Boluda, C. Ferri, F. Martínez-Plumed, J. Hernández-Orallo, and M. J. Ramírez-Quintana · 2020
Closest in time.
COCO: A platform for comparing continuous optimizers in a black-box setting
N. Hansen, A. Auger, R. Ros, O. Mersman, T. Tušar, and D. Brockhoff · 2020
Closest in time.
Towards algorithm-agnostic uncertainty estimation: Predicting classification error in an automated machine learning setting
M. König, H. H. Hoos, and J. N. van Rijn · 2020
Closest in time.
H2o automl: Scalable automatic machine learning
E. LeDell and S. Poirier · 2020
Closest in time.
Missing value imputation with mercs: A faster alternative to missforest
E. Van Wolputte and H. Blockeel · 2020
Closest in time.
Data vs classifiers, who wins?
Original
L. F. Cardoso, V. C. Santos, R. S. K. Francês, R. B. Prudêncio, and R. C. Alves · 2021
Closest in time.
Regularization is all you need: Simple neural nets can excel on tabular data
Original
A. Kadra, M. Lindauer, F. Hutter, and J. Grabocka · 2021
Closest in time.
Dynabench: Rethinking benchmarking in NLP
D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams · 2021
Closest in time.
An empirical study of moment estimators for quantile approximation
R. Mitchell, E. Frank, and G. Holmes · 2021
Closest in time.
Replacing the ex-def baseline in automl by naive automl
F. Mohr and M. Wever · 2021
Closest in time.
Personalized benchmarking with the ludwig benchmarking toolkit
A. Narayan, P. Molino, K. Goel, W. Neiswanger, and C. Re · 2021
Closest in time.
FLAML: A fast and lightweight automl library
C. Wang, Q. Wu, M. Weimer, and E. Zhu · 2021
Closest in time.
Auto-pytorch: Multi-fidelity metalearning for efficient and robust autodl
L. Zimmer, M. Lindauer, and F. Hutter · 2021
Closest in time.
Benchmark and survey of automated machine learning frameworks
M.-A. Zöller and M. F. Huber · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…