2020

SPECTER: Document-level Representation Learning using Citation-informed Transformers

Cohan, Arman, Feldman, Sergey, Beltagy, Iz et al.

Understand

Representation learning is a critical ingredient for natural language processing systems.

Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token- and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power.
For applications on scientific documents, such as classification and recommendation, the embeddings power strong performance on end tasks.
We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph.

Built on

A Comprehensive Survey on Graph Neural Networks
Original
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. 2019b · 1901
Earlier work this paper cites.
A context-aware citation recommendation model with bert and graph convolutional networks
Original
Chanwoo Jeong, Sion Jang, Hyuna Shin, Eunjeong Lucy Park, and Sungchul Choi. 2019 · 1903
Earlier work this paper cites.
Xlnet: Generalized autoregressive pretraining for language understanding
Original
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019 · 1906
Earlier work this paper cites.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Original
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar S. Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke S. Zettlemoyer, and Veselin Stoyanov. 2019 · 1907
Earlier work this paper cites.
A Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. 1996 · 1996
Earlier work this paper cites.
Medical Subject Headings (MeSH)
Carolyn E Lipscomb. 2000 · 2000
Earlier work this paper cites.
Distribution of information in biomedical abstracts and full-text publications
Martijn J. Schuemie, Marc Weeber, Bob J. A. Schijvenaars, Erik M. van Mulligen, C. Christiaan van der Eijk, Rob Jelier, Barend Mons, and Jan A. Kors. 2004 · 2004
Earlier work this paper cites.
Identifying a better measure of relatedness for mapping science
Richard Klavans and Kevin W. Boyack. 2006 · 2006
Earlier work this paper cites.
V-measure: A Conditional Entropy-based External Cluster Evaluation Measure
Andrew Rosenberg and Julia Hirschberg. 2007 · 2007
Earlier work this paper cites.
Collaborative filtering recommender systems
J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007 · 2007
Earlier work this paper cites.
Is searching full text more effective than searching abstracts?
Jimmy J. Lin. 2008 · 2008
Earlier work this paper cites.
The structural and content aspects of abstracts versus bodies of full text journal articles are different
K. Bretonnel Cohen, Helen L. Johnson, Karin M. Verspoor, Christophe Roeder, and Lawrence Hunter. 2010 · 2010
Earlier work this paper cites.
Software Framework for Topic Modelling with Large Corpora
Radim Řehůřek and Petr Sojka. 2010 · 2010
Earlier work this paper cites.
Scikit-learn: Machine learning in Python
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011 · 2011
Earlier work this paper cites.
Spectral networks and locally connected networks on graphs
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2014 · 2014
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Original
Diederik P. Kingma and Jimmy Ba. 2014 · 2014
Earlier work this paper cites.
Distributed Representations of Sentences and Documents
Quoc Le and Tomas Mikolov. 2014 · 2014
Earlier work this paper cites.
Accelerating t-SNE Using Tree-based Algorithms
Laurens van der Maaten. 2014 · 2014
Earlier work this paper cites.

Similar

Skip-thought vectors
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015 · 2015
Cited alongside, same era.
An Overview of Microsoft Academic Service (MAS) and Applications
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Paul Hsu, and Kuansan Wang. 2015 · 2015
Cited alongside, same era.
Doc2sent2vec: A novel two-phase approach for learning document representation
J Ganesh, Manish Gupta, and Vijay K. Varma. 2016 · 2016
Cited alongside, same era.
An empirical evaluation of doc2vec with practical insights into document embedding generation
Jey Han Lau and Timothy Baldwin. 2016 · 2016
Cited alongside, same era.
Google’s neural machine translation system: Bridging the gap between human and machine translation
Original
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016 · 2016
Cited alongside, same era.
A simple but tough-to-beat baseline for sentence embeddings
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017 · 2017
Cited alongside, same era.
Enriching word vectors with subword information
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017 · 2017
Cited alongside, same era.
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017 · 2017
Cited alongside, same era.
Neural Vector Spaces for Unsupervised Information Retrieval
Christophe Van Gysel, Maarten de Rijke, and Evangelos Kanoulas. 2017 · 2017
Cited alongside, same era.
Semi-supervised classification with graph convolutional networks
Thomas N Kipf and Max Welling. 2017 · 2017
Cited alongside, same era.
A Model of Extended Paragraph Vector for Document Categorization and Trend Analysis
Pengfei Liu, King Keung Wu, and Helen M. Meng. 2017 · 2017
Cited alongside, same era.
Cane: Context-aware network embedding for relation modeling
Cunchao Tu, Han Liu, Zhiyuan Liu, and Maosong Sun. 2017 · 2017
Cited alongside, same era.

Then

Universal Language Model Fine-tuning for Text Classification
Jeremy Howard and Sebastian Ruder. 2018 · 2018
Later among the works it cites.
Unsupervised Document Embedding with CNNs
Original
Chundi Liu, Shunan Zhao, and Maksims Volkovs. 2018 · 2018
Later among the works it cites.
Deep Contextualized Word Representations
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 · 2018
Later among the works it cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Later among the works it cites.
Improved semantic-aware network embedding with fine-grained word alignment
Dinghan Shen, Xinyuan Zhang, Ricardo Henao, and Lawrence Carin. 2018 · 2018
Later among the works it cites.
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018 · 2018
Later among the works it cites.
Word Mover’s Embedding: From Word2Vec to Document Embedding
Lingfei Wu, Ian En-Hsu Yen, Kun Xu, Fangli Xu, Avinash Balakrishnan, Pin-Yu Chen, Pradeep Ravikumar, and Michael J Witbrock. 2018 · 2018
Later among the works it cites.
From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing
Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik G. Learned-Miller, and Jaap Kamps. 2018 · 2018
Later among the works it cites.
Diffusion maps for textual network embedding
Xinyuan Zhang, Yitong Li, Dinghan Shen, and Lawrence Carin. 2018 · 2018
Later among the works it cites.
Estimating position bias without intrusive interventions
Anant K. Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Yen Li, Marc Najork, and Thorsten Joachims. 2019 · 2019
Later among the works it cites.
SciBERT: A Pretrained Language Model for Scientific Text
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019 · 2019
Later among the works it cites.
Improving textual network embedding with global attention via optimal transport
Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, and Lawrence Carin. 2019 · 2019
Later among the works it cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Later among the works it cites.
Quantifying Sex Bias in Clinical Studies at Scale With Automated Data Extraction
Sergey Feldman, Waleed Ammar, Kyle Lo, Elly Trepman, Madeleine van Zuylen, and Oren Etzioni. 2019 · 2019
Later among the works it cites.
A Scalable Hybrid Research Paper Recommender System for Microsoft Academic
Anshul Kanakia, Zhihong Shen, Darrin Eide, and Kuansan Wang. 2019 · 2019
Later among the works it cites.
DisSent: Learning Sentence Representations from Explicit Discourse Relations
Allen Nie, Erin Bennett, and Noah Goodman. 2019 · 2019
Later among the works it cites.
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Original
Nils Reimers and Iryna Gurevych. 2019 · 2019
Later among the works it cites.
Improving textual network learning with variational homophilic embeddings
Wenlin Wang, Chenyang Tao, Zhe Gan, Guoyin Wang, Liqun Chen, Xinyuan Zhang, Ruiyi Zhang, Qian Yang, Ricardo Henao, and Lawrence Carin. 2019 · 2085
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…