FAIR: Fairness-Aware Information Retrieval Evaluation

Understand

With the emerging needs of creating fairness-aware solutions for search and recommendation systems, a daunting challenge exists of evaluating such solutions.

• While many of the traditional information retrieval (IR) metrics can capture the relevance, diversity, and novelty for the utility with respect to users, they are not suitable for inferring whether the presented results are fair from the perspective of responsible information exposure.
• On the other hand, existing fairness metrics do not account for user utility or do not measure it adequately.
• To address this problem, we propose a new metric called FAIR.