An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Process

Understand

Mainstream machine learning conferences have seen a dramatic increase in the number of participants, along with a growing range of perspectives, in recent years.

• Members of the machine learning community are likely to overhear allegations ranging from randomness of acceptance decisions to institutional bias.
• In this work, we critically analyze the review process through a comprehensive study of papers submitted to ICLR between 2017 and 2020.
• We quantify reproducibility/randomness in review scores and acceptance decisions, and examine whether scores correlate with paper impact.