Fetching the paper…
Reading the bibliography…
This paper introduces diffusion protein language model (DPLM), a versatile protein language model that demonstrates strong generative and predictive capabilities for protein sequences.
Cath–a hierarchic classification of protein domain structures
Orengo, C. A., Michie, A. D., Jones, S., Jones, D. T., Swindells, M. B., and Thornton, J. M · 1997
Earlier work this paper cites.
A neural probabilistic language model
Bengio, Y., Ducharme, R., and Vincent, P · 2000
Earlier work this paper cites.
The protein data bank
Berman, H. M., Westbrook, J., Feng, Z., Gilliland, G., Bhat, T. N., Weissig, H., Shindyalov, I. N., and Bourne, P. E · 2000
Earlier work this paper cites.
Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., and Bottou, L · 2010
Earlier work this paper cites.
Efficient estimation of word representations in vector space
Mikolov, T., Chen, K., Corrado, G., and Dean, J · 2013
Earlier work this paper cites.
Sequence to sequence learning with neural networks
Sutskever, I., Vinyals, O., and Le, Q. V · 2014
Earlier work this paper cites.
Deep unsupervised learning using nonequilibrium thermodynamics
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S · 2015
Earlier work this paper cites.
Uniref clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., and Consortium, U · 2015
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
Non-autoregressive neural machine translation
Gu, J., Bradbury, J., Xiong, C., Li, V. O., and Socher, R · 2018
Earlier work this paper cites.
Deep contextualized word representations
Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al · 2018
Earlier work this paper cites.
BERT: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K · 2019
Earlier work this paper cites.
Mask-predict: Parallel decoding of conditional masked language models
Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L · 2019
Earlier work this paper cites.
Generative models for graph-based protein design
Ingraham, J., Garg, V., Barzilay, R., and Jaakkola, T · 2019
Earlier work this paper cites.
Roberta: A robustly optimized bert pretraining approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al · 2019
Earlier work this paper cites.
Evaluating protein transfer learning with tape
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, P., Canny, J., Abbeel, P., and Song, Y · 2019
Earlier work this paper cites.
Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., and Fergus, R · 2019
Earlier work this paper cites.
Generative modeling by estimating gradients of the data distribution
Song, Y. and Ermon, S · 2019
Earlier work this paper cites.
BERT has a mouth, and it must speak: BERT as a Markov random field language model
Wang, A. and Cho, K · 2019
Earlier work this paper cites.
Machine-learning-guided directed evolution for protein engineering
Yang, K. K., Wu, Z., and Arnold, F. H · 2019
Earlier work this paper cites.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Earlier work this paper cites.
Denoising diffusion probabilistic models
Ho, J., Jain, A., and Abbeel, P · 2020
Earlier work this paper cites.
Learning from protein structure with geometric vector perceptrons
Jing, B., Eismann, S., Suriana, P., Townshend, R. J. L., and Dror, R · 2020
Earlier work this paper cites.
Scaling laws for neural language models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D · 2020
Earlier work this paper cites.
Self-supervised contrastive learning of protein representations by mutual information maximization
Lu, A. X., Zhang, H., Ghassemi, M., and Moses, A · 2020
Earlier work this paper cites.
Transforming the language of life: transformer neural networks for protein prediction tasks
Nambiar, A., Heflin, M., Liu, S., Maslov, S., Hopkins, M., and Ritz, A · 2020
Earlier work this paper cites.
Score-based generative modeling through stochastic differential equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B · 2020
Earlier work this paper cites.
Udsmprot: universal deep sequence models for protein classification
Strodthoff, N., Wagner, P., Wenzel, M., and Samek, W · 2020
Earlier work this paper cites.
Profile prediction: An alignment-based pre-training task for protein sequence models
Sturmfels, P., Vig, J., Madani, A., and Rajani, N. F · 2020
Earlier work this paper cites.
Structured denoising diffusion models in discrete state-spaces
Austin, J., Johnson, D. D., Ho, J., Tarlow, D., and van den Berg, R · 2021
Earlier work this paper cites.
Flip: Benchmark tasks in fitness landscape inference for proteins
Dallago, C., Mou, J., Johnston, K. E., Wittmann, B. J., Bhattacharya, N., Goldman, S., Madani, A., and Yang, K. K · 2021
Earlier work this paper cites.
Prottrans: Toward understanding the language of life through self-supervised learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al · 2021
Earlier work this paper cites.
Fully non-autoregressive neural machine translation: Tricks of the trade
Gu, J. and Kong, X · 2021
Earlier work this paper cites.
Pre-training co-evolutionary protein representation via a pairwise masked language model
He, L., Zhang, S., Wu, L., Xia, H., Ju, F., Zhang, H., Liu, S., Xia, Y., Zhu, J., Deng, P., et al · 2021
Earlier work this paper cites.
Classifier-free diffusion guidance
Ho, J. and Salimans, T · 2021
Earlier work this paper cites.
Generating novel protein sequences using gibbs sampling of masked language models
Johnson, S. R., Monaco, S., Massie, K., and Syed, Z · 2021
Cited alongside, same era.
Highly accurate protein structure prediction with alphafold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al · 2021
Cited alongside, same era.
Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech
Kim, J., Kong, J., and Son, J · 2021
Cited alongside, same era.
Deep neural language modeling enables functional protein generation across families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos Jr, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al · 2021
Cited alongside, same era.
Adversarial contrastive pre-training for protein sequences
McDermott, M., Yap, B., Hsu, H., Jin, D., and Szolovits, P · 2021
Cited alongside, same era.
Diff-glat: Diffusion glancing transformer for parallel sequence to sequence learning
Qian, L., Wang, M., Liu, Y., and Zhou, H · 2022
Later among the works it cites.
Multitask prompted training enables zero-shot task generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Le Scao, T., Raja, A., et al · 2022
Later among the works it cites.
Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem
Trippe, B. L., Yim, J., Tischer, D., Baker, D., Broderick, T., Barzilay, R., and Jaakkola, T · 2022
Later among the works it cites.
Learning functional properties of proteins with language models
Unsal, S., Atas, H., Albayrak, M., Turhan, K., Acar, A. C., and Doğan, T · 2022
Later among the works it cites.
Language models generalize beyond natural proteins
Verkuil, R., Kabeli, O., Du, Y., Wicky, B. I., Milles, L. F., Dauparas, J., Baker, D., Ovchinnikov, S., Sercu, T., and Rives, A · 2022
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Language models enable zero-shot prediction of the effects of mutations on protein function
Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., and Rives, A · 2021
Cited alongside, same era.
Pre-training of deep bidirectional protein sequence representations with structural information
Min, S., Park, S., Kim, S., Choi, H.-S., Lee, B., and Yoon, S · 2021
Cited alongside, same era.
Tripletprot: deep representation learning of proteins based on siamese networks
Nourani, E., Asgari, E., McHardy, A. C., and Mofrad, M. R · 2021
Cited alongside, same era.
Glancing transformer for non-autoregressive neural machine translation
Qian, L., Zhou, H., Bao, Y., Wang, M., Qiu, L., Zhang, W., Yu, Y., and Li, L · 2021
Cited alongside, same era.
The volctrans glat system: Non-autoregressive translation meets wmt21
Qian, L., Zhou, Y., Zheng, Z., Zhu, Y., Lin, Z., Feng, J., Cheng, S., Li, L., Wang, M., and Zhou, H · 2021
Cited alongside, same era.
Msa transformer
Rao, R. M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A · 2021
Cited alongside, same era.
High-resolution image synthesis with latent diffusion models, 2021
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B · 2021
Cited alongside, same era.
Later among the works it cites.
Digress: Discrete denoising diffusion for graph generation
Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., and Frossard, P · 2022
Later among the works it cites.
Xlm-d: Decorate cross-lingual pre-training model as non-autoregressive neural machine translation
Wang, Y., He, S., Chen, G., Chen, Y., and Jiang, D · 2022
Later among the works it cites.
High-resolution de novo structure prediction from primary sequence
Wu, R., Ding, F., Wang, R., Shen, R., Zhang, X., Luo, S., Su, C., Wu, Z., Xie, Q., Berger, B., et al · 2022
Later among the works it cites.
Peer: a comprehensive and multi-task benchmark for protein sequence understanding
Xu, M., Zhang, Z., Lu, J., Zhu, Z., Zhang, Y., Chang, M., Liu, R., and Tang, J · 2022
Later among the works it cites.
Convolutions are competitive with transformers for protein sequence pretraining
Yang, K. K., Lu, A. X., and Fusi, N · 2022
Later among the works it cites.
Masked inverse folding with sequence transfer for protein representation learning
Yang, K. K., Zanichelli, N., and Yeh, H · 2022
Later among the works it cites.
Seqdiffuseq: Text diffusion with encoder-decoder transformers
Yuan, H., Yuan, Z., Tan, C., Huang, F., and Huang, S · 2022
Later among the works it cites.
Glm-130b: An open bilingual pre-trained model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al · 2022
Later among the works it cites.
Protein generation with evolutionary diffusion: sequence is all you need
Alamdari, S., Thakkar, N., van den Berg, R., Lu, A. X., Fusi, N., Amini, A. P., and Yang, K. K · 2023
Later among the works it cites.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P · 2023
Later among the works it cites.
Atomic context-conditioned protein sequence design using ligandmpnn
Dauparas, J., Lee, G. R., Pecoraro, R., An, L., Anishchenko, I., Glasscock, C., and Baker, D · 2023
Later among the works it cites.
Performance and structural coverage of the latest, in-development alphafold model
DeepMind, G · 2023
Later among the works it cites.
Specializing smaller language models towards multi-step reasoning
Fu, Y., Peng, H., Ou, L., Sabharwal, A., and Khot, T · 2023
Later among the works it cites.
Diffusionbert: Improving generative masked language models with diffusion models
He, Z., Sun, T., Wang, K., Huang, X., and Qiu, X · 2023
Later among the works it cites.
Directed acyclic transformer pre-training for high-quality non-autoregressive text generation
Huang, F., Ke, P., and Huang, M · 2023
Later among the works it cites.
Illuminating protein space with a programmable generative model
Ingraham, J. B., Baranov, M., Costello, Z., Barber, K. W., Wang, W., Ismail, A., Frappier, V., Lord, D. M., Ng-Thow-Hing, C., Van Vlack, E. R., et al · 2023
Later among the works it cites.
Generalized biomolecular modeling and design with rosettafold all-atom
Krishna, R., Wang, J., Ahern, W., Sturmfels, P., Venkatesh, P., Kalvet, I., Lee, G. R., Morey-Burrows, F. S., Anishchenko, I., Humphreys, I. R., et al · 2023
Later among the works it cites.
Lin, Y. and AlQuraishi, M · 2023
Later among the works it cites.
Joint generation of protein sequence and structure with rosettafold sequence space diffusion
Lisanza, S. L., Gershon, J. M., Tipps, S. W. K., Arnoldt, L., Hendel, S., Sims, J. N., Li, X., and Baker, D · 2023
Later among the works it cites.
Scaling data-constrained language models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C · 2023
Later among the works it cites.
Gpt-4 technical report, 2023
OpenAI · 2023
Later among the works it cites.
Saprot: Protein language modeling with structure-aware vocabulary
Su, J., Han, C., Zhou, Y., Shan, J., Zhou, X., and Yuan, F · 2023
Later among the works it cites.
Stanford alpaca: An instruction-following llama model
Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B · 2023
Later among the works it cites.
Fast and accurate protein structure search with foldseek
van Kempen, M., Kim, S. S., Tumescheit, C., Mirdita, M., Lee, J., Gilchrist, C. L., Söding, J., and Steinegger, M · 2023
Later among the works it cites.
De novo design of protein structure and function with rfdiffusion
Watson, J. L., Juergens, D., Bennett, N. R., Trippe, B. L., Yim, J., Eisenach, H. E., Ahern, W., Borst, A. J., Ragotte, R. J., Milles, L. F., et al · 2023
Later among the works it cites.
Should you mask 15% in masked language modeling?
Wettig, A., Gao, T., Zhong, Z., and Chen, D · 2023
Later among the works it cites.
Ar-diffusion: Auto-regressive diffusion model for text generation
Wu, T., Fan, Z., Liu, X., Gong, Y., Shen, Y., Jiao, J., Zheng, H.-T., Li, J., Wei, Z., Guo, J., et al · 2023
Later among the works it cites.
Graph denoising diffusion for inverse protein folding
Yi, K., Zhou, B., Shen, Y., Lio, P., and Wang, Y. G · 2023
Later among the works it cites.
Se (3) diffusion model with application to protein backbone generation
Yim, J., Trippe, B. L., De Bortoli, V., Mathieu, E., Doucet, A., Barzilay, R., and Jaakkola, T · 2023
Later among the works it cites.
Diffusion on language model embeddings for protein sequence generation
Meshchaninov, V., Strashnov, P., Shevtsov, A., Nikolaev, F., Ivanisenko, N., Kardymon, O., and Vetrov, D · 2024
Closest in time.
Direct preference optimization: Your language model is secretly a reward model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C · 2024
Closest in time.