RobustBench: a standardized adversarial robustness benchmark

Understand

As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in training robust models.

• A key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation.
• Our goal is to establish a standardized benchmark of adversarial robustness, which as accurately as possible reflects the robustness of the considered models within a reasonable computational budget.
• To this end, we start by considering the image classification task and introduce restrictions (possibly loosened in the future) on the allowed models.