Fetching the paper…
Reading the bibliography…
Protein representation learning has primarily benefited from the remarkable development of language models (LMs).
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Evaluation and improvement of multiple sequence methods for protein secondary structure prediction
James A Cuff and Geoffrey J Barton · 1999
Earlier work this paper cites.
Gene Ontology: tool for the unification of biology
Michael Ashburner, Catherine A Ball, Judith A Blake, David Botstein, Heather Butler, J Michael Cherry, Allan P Davis, Kara Dolinski, Selina S Dwight, Janan T Eppig, et al · 2000
Earlier work this paper cites.
Uniref: comprehensive and non-redundant uniprot reference clusters
Baris E Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu · 2007
Earlier work this paper cites.
SKEMPI: a structural kinetic and energetic database of mutant protein interactions and its use in empirical models
Iain H Moal and Juan Fernández-Recio · 2012
Earlier work this paper cites.
Translating embeddings for modeling multi-relational data
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko · 2013
Earlier work this paper cites.
Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Baris E Suzek, Yuqi Wang, Hongzhan Huang, Peter B McGarvey, Cathy H Wu, and UniProt Consortium · 2015
Earlier work this paper cites.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Earlier work this paper cites.
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2016
Earlier work this paper cites.
Representation learning of knowledge graphs with entity descriptions
Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun · 2016
Earlier work this paper cites.
Global analysis of protein folding using massively parallel design, synthesis, and testing
Gabriel J Rocklin, Tamuka M Chidyausiku, Inna Goreshnik, Alex Ford, Scott Houliston, Alexander Lemak, Lauren Carter, Rashmi Ravichandran, Vikram K Mulligan, Aaron Chevalier, et al · 2017
Earlier work this paper cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Learning protein sequence embeddings using information from structure
Tristan Bepler and Bonnie Berger · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · 2018
Earlier work this paper cites.
Predicting protein–protein interactions through sequence-based deep learning
Somaye Hashemifar, Behnam Neyshabur, Aly A Khan, and Jinbo Xu · 2018
Earlier work this paper cites.
DeepSF: deep convolutional neural network for mapping protein sequences to folds
Jie Hou, Badri Adhikari, and Jianlin Cheng · 2018
Earlier work this paper cites.
Deep neural network based predictions of protein interactions using primary sequences
Hang Li, Xiu-Jun Gong, Hua Yu, and Chang Zhou · 2018
Earlier work this paper cites.
Unified rational protein engineering with sequence-based deep representation learning
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church · 2019
Earlier work this paper cites.
ProteinNet: a standardized data set for machine learning of protein structure
Mohammed AlQuraishi · 2019
Earlier work this paper cites.
Multifaceted protein–protein interaction prediction based on siamese residual RCNN
Muhao Chen, Chelsea J-T Ju, Guangyu Zhou, Xuelu Chen, Tianran Zhang, Kai-Wei Chang, Carlo Zaniolo, and Wei Wang · 2019
Cited alongside, same era.
NetSurfP-2.0: Improved prediction of protein structural features by integrated deep learning
Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjaergaard Jensen, Vanessa Isabell Jurtz, Casper Kaae Soenderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, et al · 2019
Cited alongside, same era.
Linguistic knowledge and transferability of contextual representations
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith · 2019
Cited alongside, same era.
Knowledge enhanced contextual word representations
Matthew E. Peters, Mark Neumann, IV RobertL.Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith · 2019
Cited alongside, same era.
Language models as knowledge bases?
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller · 2019
Cited alongside, same era.
Probing pretrained language models for lexical semantics
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen · 2020
Later among the works it cites.
ProtTrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al · 2021
Later among the works it cites.
Domain-specific language model pretraining for biomedical natural language processing
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon · 2021
Later among the works it cites.
Highly accurate protein structure prediction with alphafold
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al · 2019
Cited alongside, same era.
Evaluating protein transfer learning with TAPE
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song · 2019
Cited alongside, same era.
XLNet: Generalized autoregressive pretraining for language understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le · 2019
Cited alongside, same era.
KG-BERT: Bert for knowledge graph completion
Liang Yao, Chengsheng Mao, and Yuan Luo · 2019
Cited alongside, same era.
ERNIE: Enhanced language representation with informative entities
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu · 2019
Cited alongside, same era.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Cited alongside, same era.
BERT-MK: Integrating graph contextualized knowledge into pre-trained language models
Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu · 2020
Cited alongside, same era.
Learning Unknown from Correlations: Graph neural network for inter-novel-protein interaction prediction
Guofeng Lv, Zhiqiang Hu, Yanguang Bi, and Shaoting Zhang · 2021
Later among the works it cites.
ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou · 2021
Later among the works it cites.
MSA transformer
Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives · 2021
Later among the works it cites.
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al · 2021
Later among the works it cites.
Detecting formal thought disorder by deep contextualized word representations
Justyna Sarzynska-Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek · 2021
Later among the works it cites.
KEPLER: A unified model for knowledge embedding and pre-trained language representation
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang · 2021
Later among the works it cites.
ProteinBERT: A universal deep-learning model of protein sequence and function
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial · 2022
Later among the works it cites.
Structure-aware protein self-supervised learning
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou · 2022
Later among the works it cites.
ProtGPT2 is a deep unsupervised language model for protein design
Noelia Ferruz, Steffen Schmidt, and Birte Höcker · 2022
Later among the works it cites.
ProGen2: exploring the boundaries of protein language models
Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani · 2022
Later among the works it cites.
Learning functional properties of proteins with language models
Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Doğan · 2022
Later among the works it cites.
Convolutions are competitive with transformers for protein sequence pretraining
Kevin K Yang, Alex Xijie Lu, and Nicolo Fusi · 2022
Later among the works it cites.
OntoProtein: Protein pretraining with gene ontology embedding
Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen · 2022
Later among the works it cites.
Bidirectional learning for offline model-based biological sequence design
Can Chen, Yingxue Zhang, Xue Liu, and Mark Coates · 2023
Closest in time.