Fetching the paper…
Reading the bibliography…
How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we survey the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews.
Design and analysis of the NIPS 2016 review process
Shah, N., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018) · 1946
Earlier work this paper cites.
Verification of forecasts expressed in terms of probability
Brier, G. W. (1950) · 1950
Earlier work this paper cites.
A progress report on the training of probability assessors
Alpert, M. and Raiffa, H. (1982) · 1982
Earlier work this paper cites.
Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy
Efron, B. and Tibshirani, R. (1986) · 1986
Earlier work this paper cites.
The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation
Cicchetti, D. V. (1991) · 1991
Earlier work this paper cites.
Controlling the false discovery rate: A practical and powerful approach to multiple testing
Benjamini, Y. and Hochberg, Y. (1995) · 1995
Earlier work this paper cites.
A comparison of authors publishing in two groups of us medical journals
Weller, A. C. (1996) · 1996
Earlier work this paper cites.
Is peer review broken? submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. what’s wrong with peer review?
McCook, A. (2006) · 2006
Earlier work this paper cites.
Examining the value added by committee discussion in the review of applications for research awards
Obrecht, M., Tibelius, K., and D’Aloisio, G. (2007) · 2007
Earlier work this paper cites.
Perceptions of ethical problems with scientific journal peer review: an exploratory study
Resnik, D. B., Gutierrez-Ford, C., and Peddada, S. (2008) · 2008
Earlier work this paper cites.
Peer review: benefits, perceptions and alternatives
Ware, M. (2008) · 2008
Earlier work this paper cites.
Conference reviewing considered harmful
Anderson, T. (2009) · 2009
Earlier work this paper cites.
How many scientists fabricate and falsify research? a systematic review and meta-analysis of survey data
Fanelli, D. (2009) · 2009
Earlier work this paper cites.
I Hate Your Paper. Many say the peer review system is broken. Here’s how some journals are trying to fix it
Akst, J. (2010) · 2010
Earlier work this paper cites.
A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants
Bornmann, L., Mutz, R., and Daniel, H.-D. (2010) · 2010
Earlier work this paper cites.
Gender and overconfidence: Are girls really overconfident?
Dahlbom, L., Jakobsson, A., Jakobsson, N., and Kotsadam, A. (2011) · 2011
Earlier work this paper cites.
A large scale randomized controlled trial on herding in peer-review discussions
Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., and Daumé III, H. (2020) · 2011
Earlier work this paper cites.
A status-enhancement account of overconfidence
Anderson, C., Brion, S., Moore, D. A., and Kennedy, J. A. (2012) · 2012
Cited alongside, same era.
Panel discussion does not improve reliability of peer review for medical research grant proposals
Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., and Väänänen, K. (2012) · 2012
Cited alongside, same era.
Peering at the peer review process for conference submissions
Gardner, A., Willey, K., Jolly, L., and Tibbits, G. (2012) · 2012
Cited alongside, same era.
A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model
Roos, M., Rothe, J., Rudolph, J., Scheuermann, B., and Stoyan, D. (2012) · 2012
Cited alongside, same era.
Peer review in a changing world: An international study measuring the attitudes of researchers
Mulligan, A., Hall, L., and Raphael, E. (2013) · 2013
Cited alongside, same era.
Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings
Wang, J. and Shah, N. B. (2019) · 2019
Later among the works it cites.
A survey of accepted authors in computer systems conferences
Frachtenberg, E. and Koster, N. (2020) · 2020
Later among the works it cites.
Mitigating manipulation in peer review via randomized reviewer assignments
Jecmen, S., Zhang, H., Liu, R., Shah, N. B., Conitzer, V., and Fang, F. (2020) · 2020
Later among the works it cites.
Potential organized fraud in on-going asplos reviews
Vijaykumar, T. N. (2020) · 2020
Later among the works it cites.
The NeurIPS 2021 consistency experiment
Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (2021) · 2021
Later among the works it cites.
A Bayesian model for calibrating conference review scores
Ge, H., Welling, M., and Ghahramani, Z. (2013) · 2021
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cortes, C. and Lawrence, N. D. (2021) · 2014
Cited alongside, same era.
Gender gaps in overestimation of math performance
Bench, S. W., Lench, H. C., Liew, J., Miner, K. N., and Flores, S. A. (2015) · 2015
Cited alongside, same era.
Peer review: still king in the digital age
Nicholas, D., Watkinson, A., Jamali, H. R., Herman, E., Tenopir, C., Volentine, R., Allard, S., and Levine, K. (2015) · 2015
Cited alongside, same era.
Peer review in 2015 a global view
Taylor and Francis group (2015) · 2015
Cited alongside, same era.
Publishing research consortium peer review survey 2015
Ware, M. (2016) · 2015
Cited alongside, same era.
What motivates people to review articles? the case of the human-computer interaction community
Nobarany, S., Booth, K. S., and Hsieh, G. (2016) · 2016
Cited alongside, same era.
Let’s make peer review scientific
Rennie, D. (2016) · 2016
Cited alongside, same era.
Later among the works it cites.
Collusion rings threaten the integrity of computer science research
Littman, M. L. (2021) · 2021
Later among the works it cites.
Research funding randomly allocated? a survey of scientists’ views on peer review and lottery
Philipps, A. (2021) · 2021
Later among the works it cites.
You are the best reviewer of your own papers: An owner-assisted scoring mechanism
Su, W. (2021) · 2021
Later among the works it cites.
Making paper reviewing robust to bid manipulation attacks
Wu, R., Guo, C., Wu, F., Kidambi, R., van der Maaten, L., and Weinberger, K. (2021) · 2021
Later among the works it cites.
Calibration with privacy in peer review
Ding, W., Kamath, G., Wang, W., and Shah, N. B. (2022) · 2022
Closest in time.
Rethinking the funding line at the Swiss national science foundation: Bayesian ranking and lottery
Heyard, R., Ott, M., Salanti, G., and Egger, M. (2022) · 2022
Closest in time.
Tradeoffs in preventing manipulation in paper bidding for reviewer assignment
Jecmen, S., Shah, N. B., Fang, F., and Conitzer, V. (2022) · 2022
Closest in time.
To ArXiv or not to ArXiv: A study quantifying pros and cons of posting preprints online
Rastogi, C., Stelmakh, I., Shen, X., Meila, M., Echenique, F., Chawla, S., and Shah, N. (2022) · 2022
Closest in time.
An overview of challenges, experiments, and computational solutions in peer review
Shah, N. B. (2022) · 2022
Closest in time.
Cite-seeing and reviewing: A study on citation bias in peer review
Stelmakh, I., Rastogi, C., Liu, R., Chawla, S., Echenique, F., and Shah, N. B. (2022) · 2022
Closest in time.