Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms

Understand

A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of pre-defined quality targets achieved not later than a given runtime.

• In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF.
• We argue that the EAF has several advantages over the target-based ECDF.
• In particular, it does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics that enrich the analysis.