2016

COCO: Performance Assessment

Hansen, Nikolaus, Auger, Anne, Brockhoff, Dimo et al.

Understand

We present an any-time performance assessment for benchmarking numerical optimization algorithms in a black-box scenario, applied within the COCO benchmarking platform.

The performance assessment is based on runtimes measured in number of objective function evaluations to reach one or several quality indicator target values.
We argue that runtime is the only available measure with a generic, meaningful, and quantitative interpretation.
We discuss the choice of the target values, runlength-based targets, and the aggregation of results by using simulated restarts, averages, and empirical distribution functions.

Built on

S.S. Stevens (1946). On the theory of scales of measurement. Science 103(2684), pp. 677-680
1946
Earlier work this paper cites.
B. Efron and R. Tibshirani (1994). An introduction to the bootstrap . CRC Press
1994
Earlier work this paper cites.
J. N. Hooker Testing heuristics: We have it all wrong. In Journal of Heuristics, pages 33-42, 1995
1995
Earlier work this paper cites.
K. Price. Differential evolution vs. the functions of the second ICEO. In Proceedings of the IEEE International Congress on Evolutionary Computation, pages 153–157, 1997
1997
Earlier work this paper cites.
H.H. Hoos and T. Stützle. Evaluating Las Vegas algorithms—pitfalls and remedies. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (UAI-98) , pages 238–245, 1998
1998
Earlier work this paper cites.
E.D. Dolan, J. J. Moré (2002). Benchmarking optimization software with performance profiles. Mathematical Programming 91.2, 201-213
2002
Earlier work this paper cites.

Similar

A. Auger and N. Hansen. Performance evaluation of an advanced local search evolutionary algorithm. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC 2005) , pages 1777–1784, 2005
2005
Cited alongside, same era.
A. Auger, N. Hansen, J.M. Perez Zerpa, R. Ros and M. Schoenauer (2009). Empirical comparisons of several derivative free optimization algorithms. In Acte du 9ime colloque national en calcul des structures, Giens
2009
Cited alongside, same era.
N. Hansen, A. Auger, S. Finck, and R. Ros (2009). Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup, Research Report RR-6828 , Inria
2009
Cited alongside, same era.
Jorge J. Moré and Stefan M. Wild. Benchmarking Derivative-Free Optimization Algorithms, SIAM J. Optim. , 20(1), 172–191, 2009
2009
Cited alongside, same era.
N. Hansen, A. Auger, R. Ros, S. Finck, and P. Posik (2010). Comparing Results of 31 Algorithms from the Black-Box Optimization Benchmarking BBOB-2009. Workshop Proceedings of the GECCO Genetic and Evolutionary Computation Conference 2010, ACM, pp. 1689-1696
2010
Cited alongside, same era.

Then

N. Hansen, S. Finck, R. Ros, and A. Auger (2009). Real-parameter black-box optimization benchmarking 2009: Noiseless functions definitions. Research Report RR-6829 , Inria, updated February 2010
2010
Later among the works it cites.
Luis Miguel Rios and Nikolaos V Sahinidis. Derivative-free optimization: A review of algorithms and comparison of software implementations. Journal of Global Optimization, 56(3):1247– 1293, 2013
2013
Later among the works it cites.
D. Brockhoff, T. Tušar, D. Tušar, T. Wagner, N. Hansen, A. Auger, (2016). Biobjective Performance Assessment with the COCO Platform . ArXiv e-prints , arXiv:1605.01746
Original
2016
Closest in time.
N. Hansen, A. Auger, O. Mersmann, T. Tušar, D. Brockhoff (2016). COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting . ArXiv e-prints , arXiv:1603:08785
Original
2016
Closest in time.
N. Hansen, T. Tušar, A. Auger, D. Brockhoff, O. Mersmann (2016). COCO: The Experimental Procedure , ArXiv e-prints , arXiv:1603.08776
Original
2016
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…