Arbitrariness of peer review: A Bayesian analysis of the NIPS experiment

Understand

The principle of peer review is central to the evaluation of research, by ensuring that only high-quality items are funded or published.

• But peer review has also received criticism, as the selection of reviewers may introduce biases in the system.
• In 2014, the organizers of the ``Neural Information Processing Systems\rq\rq{} conference conducted an experiment in which $10\%$ of submitted manuscripts (166 items) went through the review process twice.
• Arbitrariness was measured as the conditional probability for an accepted submission to get rejected if examined by the second committee.