Dissecting Sample Hardness: A Fine-Grained Analysis of Hardness Characterization Methods for Data-Centric AI

Understand

Characterizing samples that are difficult to learn from is crucial to developing highly performant ML models.

• This has led to numerous Hardness Characterization Methods (HCMs) that aim to identify "hard" samples.
• However, there is a lack of consensus regarding the definition and evaluation of "hardness".
• Unfortunately, current HCMs have only been evaluated on specific types of hardness and often only qualitatively or with respect to downstream performance, overlooking the fundamental quantitative identification task.