2022

To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online

Rastogi, Charvi, Stelmakh, Ivan, Shen, Xinwei et al.

Understand

Double-blind conferences have engaged in debates over whether to allow authors to post their papers online on arXiv or elsewhere during the review process.

Independently, some authors of research papers face the dilemma of whether to put their papers on arXiv due to its pros and cons.
We conduct a study to substantiate this debate and dilemma via quantitative measurements.
Specifically, we conducted surveys of reviewers in two top-tier double-blind computer science conferences -- ICML 2021 (5361 submissions and 4699 reviewers) and EC 2021 (498 submissions and 190 reviewers).

Built on

A new measure of rank correlation
Kendall, M. G. (1938) · 1938
Earlier work this paper cites.
Design and analysis of the NIPS 2016 review process
Shah, N., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018) · 1946
Earlier work this paper cites.
Is there any measurable benefit in publishing preprints in the arXiv section quantitative biology?
Original
Aman, V. (2014) · 1955
Earlier work this paper cites.
The Matthew effect in science
Merton, R. K. (1968) · 1968
Earlier work this paper cites.
The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review
Blank, R. M. (1991) · 1991
Earlier work this paper cites.
The Matilda effect in science
Rossiter, M. W. (1993) · 1993
Earlier work this paper cites.
Smoothing by local regression: Principles and methods
Cleveland, W. S. and Loader, C. (1996) · 1996
Earlier work this paper cites.
Averaging correlations: Expected values and bias in combined pearson rs and fisher’s z transformations
Corey, D., Dunlap, W., and Burke, M. (1998) · 1998
Earlier work this paper cites.
US and Non-US Submissions: An Analysis of Reviewer Bias
Link, A. M. (1998) · 1998
Earlier work this paper cites.
Scientists behaving badly
Martinson, B. C., Anderson, M. S., and De Vries, R. (2005) · 2005
Earlier work this paper cites.
Impact of double-blind reviewing on SIGMOD publication rates
Madden, S. and DeWitt, D. (2006) · 2006
Earlier work this paper cites.
Single- versus Double-blind reviewing: An analysis of the literature
Snodgrass, R. (2006) · 2006
Earlier work this paper cites.
Impact of double blind reviewing on SIGMOD publication: A more detail analysis
Tung, A. K. (2006) · 2006
Earlier work this paper cites.
De-anonymization of authors through arXiv submissions during double-blind review
Original
Bharadhwaj, H., Turpin, D., Garg, A., and Anderson, A. (2020) · 2007
Earlier work this paper cites.

Similar

Why do peer reviews decline to review? A survey
Tite, L. and Schroter, S. (2007) · 2007
Cited alongside, same era.
Double-blind review favours increased representation of female authors
Budden, A. E., Tregenza, T., Aarssen, L. W., Koricheva, J., Leimu, R., and Lortie, C. J. (2008) · 2008
Cited alongside, same era.
Perceptions of ethical problems with scientific journal peer review: An exploratory study
Resnik, D. B., Gutierrez-Ford, C., and Peddada, S. (2008) · 2008
Cited alongside, same era.
How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data
Fanelli, D. (2009) · 2009
Cited alongside, same era.
A large scale randomized controlled trial on herding in peer-review discussions
Original
Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., and Daumé III, H. (2020) · 2011
Cited alongside, same era.
Saint Matthew strikes again. An agent-based model of peer review and the scientific community structure
Squazzoni, F. and Claudio, G. (2012) · 2012
Cited alongside, same era.
The Toronto Paper Matching System: An automated paper-reviewer assignment system
Charlin, L. and Zemel, R. S. (2013) · 2013
Cited alongside, same era.
The Matilda effect in science communication: An experiment on gender bias in publication quality perceptions and collaboration interest
Knobloch-Westerwick, S., Glynn, C. J., and Huge, M. (2013) · 2013
Cited alongside, same era.
By the numbers: Track record, flawed reviews, journal space, and the fate of talented authors
Thorngate, W. and Chowdhury, W. (2013) · 2013
Cited alongside, same era.
What motivates people to review articles? The case of the human-computer interaction community
Nobarany, S., Booth, K. S., and Hsieh, G. (2016) · 2016
Cited alongside, same era.
Double-blind reviewing at EvoLang 11 reveals gender bias
Roberts, S. and Verhoef, T. (2016) · 2016
Cited alongside, same era.
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., and Heavlin, W. (2017) · 2017
Cited alongside, same era.

Then

Paper matching with local fairness constraints
Kobren, A., Saha, B., and McCallum, A. (2019) · 2019
Later among the works it cites.
Mitigating manipulation in peer review via randomized reviewer assignments
Jecmen, S., Zhang, H., Liu, R., Shah, N. B., Conitzer, V., and Fang, F. (2020) · 2020
Later among the works it cites.
The NeurIPS 2021 consistency experiment
Beygelzimer, A., Dauphin, Y., Liang, P., and Wortman Vaughan, J. (2021) · 2021
Later among the works it cites.
Computer Science Rankings: Economics and Computation
CSRankings (2021) · 2021
Later among the works it cites.
Uncovering latent biases in text: Method and application to peer review
Manzoor, E. and Shah, N. B. (2021) · 2021
Later among the works it cites.
Condorcet internet voting service
Myers, A. (2003) · 2021
Later among the works it cites.
Loss functions, axioms, and peer review
Noothigattu, R., Shah, N., and Procaccia, A. (2021) · 2021
Later among the works it cites.
QS world university rankings by subject 2021: Computer Science and Information Systems
QS (2021a) · 2021
Later among the works it cites.
QS world university rankings by subject 2021: Economics & Econometrics
QS (2021b) · 2021
Later among the works it cites.
An overview of challenges, experiments, and computational solutions in peer review
Shah, N. B. (2021) · 2021
Later among the works it cites.
Does double-blind peer review reduce bias? Evidence from a top computer science conference
Sun, M., Barry Danfa, J., and Teplitskiy, M. (2021) · 2021
Later among the works it cites.
Is preprint the future of science? A thirty year journey of online preprint services
Original
Xie, B., Shen, Z., and Wang, K. (2021) · 2021
Later among the works it cites.
ICML 2020. Comprehensive analysis of authors, organizations, and countries
Ivanov, S. (2020) · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…