Loss Functions, Axioms, and Peer Review

Understand

It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper.

• More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review.
• In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping.
• The key challenge that arises is the specification of a loss function for ERM.