On Evaluating Explanation Utility for Human-AI Decision Making in NLP

Understand

Is explainability a false promise? This debate has emerged from the insufficient evidence that explanations help people in situations they are introduced for.

• More human-centered, application-grounded evaluations of explanations are needed to settle this.
• Yet, with no established guidelines for such studies in NLP, researchers accustomed to standardized proxy evaluations must discover appropriate measurements, tasks, datasets, and sensible models for human-AI teams in their studies.
• To aid with this, we first review existing metrics suitable for application-grounded evaluation.