Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair

Understand

More capable language models increasingly saturate existing task benchmarks, in some cases outperforming humans.

• This has left little headroom with which to measure further progress.
• Adversarial dataset creation has been proposed as a strategy to construct more challenging datasets, and two common approaches are: (1) filtering out easy examples and (2) model-in-the-loop data collection.
• In this work, we study the impact of applying each approach to create more challenging evaluation datasets.