Evaluating Machine Learning Models with NERO: Non-Equivariance Revealed on Orbits

Understand

Proper evaluations are crucial for better understanding, troubleshooting, interpreting model behaviors and further improving model performance.

• While using scalar-based error metrics provides a fast way to overview model performance, they are often too abstract to display certain weak spots and lack information regarding important model properties, such as robustness.
• This not only hinders machine learning models from being more interpretable and gaining trust, but also can be misleading to both model developers and users.
• Additionally, conventional evaluation procedures often leave researchers unclear about where and how model fails, which complicates model comparisons and further developments.