On the Utility of Prediction Sets in Human-AI Teams

Understand

Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation.

• Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels.
• We explore how such prediction sets impact expert decision-making in human-AI teams.
• Our evaluation on human subjects finds that set valued predictions positively impact experts.