Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking

Understand

We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform.

• Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset.
• Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP.
• This allows users to interact with uploaded models in real time to assess their quality, and permits the collection of additional metrics such as memory use, throughput, and robustness, which -- despite their importance to practitioners -- have traditionally been absent from leaderboards.