2021

Can We Automate Scientific Reviewing?

Yuan, Weizhe, Liu, Pengfei, Neubig, Graham

Understand

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications.

At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts.
Thus, providing high-quality reviews of this growing number of papers is a significant challenge.
In this work, we ask the question "can we automate scientific reviewing?", discussing the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers.

Built on

Scibert: A pretrained language model for scientific text
Original
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019 · 1903
Earlier work this paper cites.
Bertscore: Evaluating text generation with bert
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019 · 1904
Earlier work this paper cites.
On extractive and abstractive neural document summarization with transformer language models
Original
Sandeep Subramanian, Raymond Li, Jonathan Pilault, and Christopher Pal. 2019 · 1909
Earlier work this paper cites.
Extractive summarization of long documents by combining global and local context
Original
Wen Xiao and Giuseppe Carenini. 2019 · 1909
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 1910
Earlier work this paper cites.
Compressive transformers for long-range sequence modelling
Original
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and T. Lillicrap. 2020 · 1911
Earlier work this paper cites.
Bootstrap methods: another look at the jackknife
Bradley Efron. 1992 · 1992
Earlier work this paper cites.
Literature dynamics: Studies on growth, diffusion, and epidemics
Albert N Tabah. 1999 · 1999
Earlier work this paper cites.
Automatic evaluation of summaries using n-gram co-occurrence statistics
Chin-Yew Lin and Eduard Hovy. 2003 · 2003
Earlier work this paper cites.
Tldr: Extreme summarization of scientific documents
Original
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld. 2020b · 2004
Earlier work this paper cites.
Language models are few-shot learners
Original
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 2005
Earlier work this paper cites.
Aspect-based sentiment analysis of scientific reviews
Original
Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020 · 2006
Earlier work this paper cites.
Mitigating manipulation in peer review via randomized reviewer assignments
Original
Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B Shah, Vincent Conitzer, and Fei Fang. 2020 · 2006
Earlier work this paper cites.
Peer review: A flawed process at the heart of science and journals
R. Smith. 2006 · 2006
Earlier work this paper cites.
Scientific paper summarization using citation summary networks
Vahed Qazvinian and Dragomir R. Radev. 2008 · 2008
Earlier work this paper cites.
Bibliometrics and citation analysis: from the science citation index to cybermetrics
Nicola De Bellis. 2009 · 2009
Earlier work this paper cites.
Using citations to generate surveys of scientific paradigms
Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishan, Vahed Qazvinian, Dragomir Radev, and David Zajic. 2009 · 2009
Earlier work this paper cites.
Gsum: A general framework for guided neural abstractive summarization
Original
Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020 · 2010
Earlier work this paper cites.
Paper gestalt
Carven Von Bearnensquash. 2010 · 2010
Earlier work this paper cites.
Automatically predicting peer-review helpfulness
Wenting Xiong and Diane Litman. 2011 · 2011
Earlier work this paper cites.
A system for summarizing scientific topics starting from keywords
Rahul Jha, Amjad Abu-Jbara, and Dragomir Radev. 2013 · 2013
Earlier work this paper cites.
The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning
Reuven Y Rubinstein and Dirk P Kroese. 2013 · 2013
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba. 2014 · 2014
Earlier work this paper cites.

Similar

Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references
Lutz Bornmann and Rüdiger Mutz. 2015 · 2015
Cited alongside, same era.
Teaching machines to read and comprehend
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015 · 2015
Cited alongside, same era.
Surveyor: A system for generating coherent survey articles for scientific topics
Rahul Jha, Reed Coke, and Dragomir R. Radev. 2015a · 2015
Cited alongside, same era.
The arbitrariness of reviews, and advice for school administrators
John Langford and Mark Guzdial. 2015 · 2015
Cited alongside, same era.
Your paper has been accepted, rejected, or whatever: Automatic generation of scientific paper reviews
Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. 2016 · 2016
Cited alongside, same era.
Man is to computer programmer as woman is to homemaker? debiasing word embeddings
Original
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016 · 2016
Cited alongside, same era.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 · 2016
Cited alongside, same era.
Scientific article summarization using citation-context and article’s discourse structure
Original
Arman Cohan and Nazli Goharian. 2017 · 2017
Cited alongside, same era.
Unsupervised query-focused multi-document summarization using the cross entropy method
Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017 · 2017
Cited alongside, same era.
Integrating the trend of research interest for reviewer assignment
Jian Jin, Qian Geng, Qian Zhao, and Lixue Zhang. 2017 · 2017
Cited alongside, same era.
Summarunner: A recurrent neural network based sequence model for extractive summarization of documents
Original
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017 · 2017
Cited alongside, same era.
A deep reinforced model for abstractive summarization
Original
Romain Paulus, Caiming Xiong, and Richard Socher. 2017 · 2017
Cited alongside, same era.

Then

Modularized and attention-based recurrent convolutional neural network for automatic academic paper aspect scoring
Feng Qiao, Lizhen Xu, and Xiaowei Han. 2018 · 2018
Later among the works it cites.
Sentiment analysis of peer review texts for scholarly papers
Ke Wang and Xiaojun Wan. 2018 · 2018
Later among the works it cites.
Gender bias in coreference resolution: Evaluation and debiasing methods
Original
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018 · 2018
Later among the works it cites.
PaRe: A paper-reviewer matching approach using a common topic space
Omer Anjum, Hongyu Gong, Suma Bhat, Wen-Mei Hwu, and JinJun Xiong. 2019 · 2019
Later among the works it cites.
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
A summarization system for scientific documents
Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, O. Nakash, O. Boni, Haggai Roitman, Doron Cohen, B. Weiner, Y. Mass, Or Rivlin, G. Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, F. Bonin, and D. Konopnicki. 2019 · 2019
Later among the works it cites.
Inducing document structure for aspect-based summarization
Lea Frermann and Alexandre Klementiev. 2019 · 2019
Later among the works it cites.
Does my rebuttal matter? insights from a major NLP conference
Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019 · 2019
Later among the works it cites.
Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019 · 2019
Later among the works it cites.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019 · 2019
Later among the works it cites.
Evaluating gender bias in machine translation
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019 · 2019
Later among the works it cites.
On testing for biases in peer review
Ivan Stelmakh, Nihar Shah, and Aarti Singh. 2019 · 2019
Later among the works it cites.
Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks
Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R. Fabbri, Irene Li, Dan Friedman, and Dragomir R. Radev. 2019b · 2019
Later among the works it cites.
Writing strategies for science communication: Data and computational analysis
Tal August, Lauren Kim, Katharina Reinecke, and Noah A. Smith. 2020 · 2020
Later among the works it cites.
TLDR: Extreme summarization of scientific documents
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. 2020a · 2020
Later among the works it cites.
Wikiasp: A dataset for multi-domain aspect-based summarization
Original
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2020 · 2020
Later among the works it cites.
SciREX: A challenge dataset for document-level information extraction
Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020 · 2020
Later among the works it cites.
S2ORC: The semantic scholar open research corpus
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020 · 2020
Later among the works it cites.
Uncovering latent biases in text: Method and application to peer review
Original
Emaad Manzoor and Nihar B. Shah. 2020 · 2020
Later among the works it cites.
What can we do to improve peer review in NLP?
Anna Rogers and Isabelle Augenstein. 2020 · 2020
Later among the works it cites.
Fact or fiction: Verifying scientific claims
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020 · 2020
Later among the works it cites.
Reviewrobot: Explainable paper review generation based on knowledge synthesis
Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020 · 2020
Later among the works it cites.
Automatic generation of citation texts in scholarly papers: A pilot study
Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020 · 2020
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…