Measuring and signing fairness as performance under multiple stakeholder distributions

Understand

As learning machines increase their influence on decisions concerning human lives, analyzing their fairness properties becomes a subject of central importance.

• Yet, our best tools for measuring the fairness of learning systems are rigid fairness metrics encapsulated as mathematical one-liners, offer limited power to the stakeholders involved in the prediction task, and are easy to manipulate when we exhort excessive pressure to optimize them.
• To advance these issues, we propose to shift focus from shaping fairness metrics to curating the distributions of examples under which these are computed.
• In particular, we posit that every claim about fairness should be immediately followed by the tagline "Fair under what examples, and collected by whom?".