Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches

Understand

Developing state-of-the-art approaches for specific tasks is a major driving force in our research community.

• Depending on the prestige of the task, publishing it can come along with a lot of visibility.
• The question arises how reliable are our evaluation methodologies to compare approaches? One common methodology to identify the state-of-the-art is to partition data into a train, a development and a test set.
• Researchers can train and tune their approach on some part of the dataset and then select the model that worked best on the development set for a final evaluation on unseen test data.