Estimating Example Difficulty Using Variance of Gradients

Understand

In machine learning, a question of great interest is understanding what examples are challenging for a model to classify.

• Identifying atypical examples ensures the safe deployment of models, isolates samples that require further human inspection and provides interpretability into model behavior.
• In this work, we propose Variance of Gradients (VoG) as a valuable and efficient metric to rank data by difficulty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing.
• We show that data points with high VoG scores are far more difficult for the model to learn and over-index on corrupted or memorized examples.