A critical analysis of metrics used for measuring progress in artificial intelligence

Understand

Comparing model performances on benchmark datasets is an integral part of measuring and driving progress in artificial intelligence.

• A model's performance on a benchmark dataset is commonly assessed based on a single or a small set of performance metrics.
• While this enables quick comparisons, it may entail the risk of inadequately reflecting model performance if the metric does not sufficiently cover all performance characteristics.
• It is unknown to what extent this might impact benchmarking efforts.