Transformers and Large Language Models for Chemistry and Drug Discovery

Yin, X., Chen, W., Wu, X., Yue, H.: Fine-tuning and visualization of convolutional neural networks. In: 2017 12th IEEE Conference on Industrial Electronics and Applications (ICIEA), pp. 1310–1315 (2017). https://doi.org/10.1109/ICIEA.2017.8283041 . ISSN: 2158-2297

2017

Earlier work this paper cites.

Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J.Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D.E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P.W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X.L., Li, X., Ma, T., Malik, A., Manning, C.D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J.C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J.S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A.W., Tramèr, F., Wang, R.E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S.M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., Liang, P.: On the Opportunities and Risks of Foundation Models. arXiv. arXiv:2108.07258 [cs] (2022). http://arxiv.org/abs/2108.07258 Accessed 2023-06-27

Original

2022

Earlier work this paper cites.

Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate. arXiv. arXiv:1409.0473 [cs, stat] (2016). https://doi.org/10.48550/arXiv.1409.0473 . http://arxiv.org/abs/1409.0473 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Cho, K., Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv. arXiv:1406.1078 [cs, stat] (2014). https://doi.org/10.48550/arXiv.1406.1078 . http://arxiv.org/abs/1406.1078 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Hochreiter, S., Schmidhuber, J.: Long Short-Term Memory. Neural Computation 9

2023

Earlier work this paper cites.

Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to Sequence Learning with Neural Networks. arXiv. arXiv:1409.3215 [cs] (2014). https://doi.org/10.48550/arXiv.1409.3215 . http://arxiv.org/abs/1409.3215 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need. arXiv. arXiv:1706.03762 [cs] (2017). https://doi.org/10.48550/arXiv.1706.03762 . http://arxiv.org/abs/1706.03762 Accessed 2023-02-28

Original

2023

Earlier work this paper cites.

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y.T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M.T., Zhang, Y.: Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv. arXiv:2303.12712 [cs] (2023). http://arxiv.org/abs/2303.12712 Accessed 2023-03-30

Original

2023

Earlier work this paper cites.

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., Leahy, C.: The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv. arXiv:2101.00027 [cs] (2020). https://doi.org/10.48550/arXiv.2101.00027 . http://arxiv.org/abs/2101.00027 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv. arXiv:1910.10683 [cs, stat] (2020). https://doi.org/10.48550/arXiv.1910.10683 . http://arxiv.org/abs/1910.10683 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. arXiv:1810.04805 [cs] (2019). https://doi.org/10.48550/arXiv.1810.04805 . http://arxiv.org/abs/1810.04805 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv. arXiv:1908.10084 [cs] (2019). https://doi.org/10.48550/arXiv.1908.10084 . http://arxiv.org/abs/1908.10084 Accessed 2023-02-21

Original

2023

Earlier work this paper cites.

Liu, P.J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., Shazeer, N.: Generating Wikipedia by Summarizing Long Sequences. arXiv. arXiv:1801.10198 [cs] (2018). http://arxiv.org/abs/1801.10198 Accessed 2023-02-06

Original

2023

Earlier work this paper cites.

Kryscinski, W., Keskar, N.S., McCann, B., Xiong, C., Socher, R.: Neural Text Summarization: A Critical Evaluation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 540–551. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1051 . https://www.aclweb.org/anthology/D19-1051

2023

Earlier work this paper cites.

OpenAI: GPT-4 Technical Report. arXiv. arXiv:2303.08774 [cs] (2023). https://doi.org/10.48550/arXiv.2303.08774 . http://arxiv.org/abs/2303.08774 Accessed 2023-07-27

Original

2023

Earlier work this paper cites.

Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S.A.A., Ballard, A.J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A.W., Kavukcuoglu, K., Kohli, P., Hassabis, D.: Highly accurate protein structure prediction with AlphaFold. Nature 596

2023

Earlier work this paper cites.

Yang, Z., Zeng, X., Zhao, Y., Chen, R.: AlphaFold2 and its applications in the fields of biology and medicine. Signal Transduction and Targeted Therapy 8

2023

Earlier work this paper cites.

Akdel, M., Pires, D.E.V., Pardo, E.P., Jänes, J., Zalevsky, A.O., Mészáros, B., Bryant, P., Good, L.L., Laskowski, R.A., Pozzati, G., Shenoy, A., Zhu, W., Kundrotas, P., Serra, V.R., Rodrigues, C.H.M., Dunham, A.S., Burke, D., Borkakoti, N., Velankar, S., Frost, A., Basquin, J., Lindorff-Larsen, K., Bateman, A., Kajava, A.V., Valencia, A., Ovchinnikov, S., Durairaj, J., Ascher, D.B., Thornton, J.M., Davey, N.E., Stein, A., Elofsson, A., Croll, T.I., Beltrao, P.: A structural biology community assessment of AlphaFold2 applications. Nature Structural & Molecular Biology 29

2023

Earlier work this paper cites.

Huang, K., Fu, T., Gao, W., Zhao, Y., Roohani, Y., Leskovec, J., Coley, C., Xiao, C., Sun, J., Zitnik, M.: Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1

2023

Earlier work this paper cites.

Kearnes, S.M., Maser, M.R., Wleklinski, M., Kast, A., Doyle, A.G., Dreher, S.D., Hawkins, J.M., Jensen, K.F., Coley, C.W.: The Open Reaction Database. Journal of the American Chemical Society 143

2023

Earlier work this paper cites.

Lowe, D.: Extraction of chemical structures and reactions from the literature. PhD thesis, University of Cambridge (2012). doi:10.17863/CAM.16293 Accessed 2023-07-27

2023

Earlier work this paper cites.

Wu, Z., Ramsundar, B., Feinberg, E.N., Gomes, J., Geniesse, C., Pappu, A.S., Leswing, K., Pande, V.: MoleculeNet: a benchmark for molecular machine learning. Chemical Science 9

2023

Earlier work this paper cites.

Cadeddu, A., Wylie, E.K., Jurczak, J., Wampler-Doty, M., Grzybowski, B.A.: Organic Chemistry as a Language and the Implications of Chemical Linguistics for Structural and Retrosynthetic Analyses. Angewandte Chemie International Edition 53

2023

Earlier work this paper cites.

Wołos, A., Koszelewski, D., Roszak, R., Szymkuć, S., Moskal, M., Ostaszewski, R., Herrera, B.T., Maier, J.M., Brezicki, G., Samuel, J., Lummiss, J.A.M., McQuade, D.T., Rogers, L., Grzybowski, B.A.: Computer-designed repurposing of chemical wastes into drugs. Nature 604

2023

Earlier work this paper cites.

Schwaller, P., Hoover, B., Reymond, J.-L., Strobelt, H., Laino, T.: Extraction of organic chemistry grammar from unsupervised learning of chemical reactions. Science Advances 7

2023

Earlier work this paper cites.

Weininger, D.: SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences 28

2023

Earlier work this paper cites.

O’Boyle, N., Dalke, A.: DeepSMILES: An Adaptation of SMILES for Use in Machine-Learning of Chemical Structures. ChemRxiv (2018). https://doi.org/10.26434/chemrxiv.7097960.v1 . https://chemrxiv.org/engage/chemrxiv/article-details/60c73ed6567dfe7e5fec388d Accessed 2023-07-27

2023

Earlier work this paper cites.

Krenn, M., Häse, F., Nigam, A., Friederich, P., Aspuru-Guzik, A.: Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation. Machine Learning: Science and Technology 1

2023

Earlier work this paper cites.

Krenn, M., Ai, Q., Barthel, S., Carson, N., Frei, A., Frey, N.C., Friederich, P., Gaudin, T., Gayle, A.A., Jablonka, K.M., Lameiro, R.F., Lemm, D., Lo, A., Moosavi, S.M., Nápoles-Duarte, J.M., Nigam, A., Pollice, R., Rajan, K., Schatzschneider, U., Schwaller, P., Skreta, M., Smit, B., Strieth-Kalthoff, F., Sun, C., Tom, G., Rudorff, G.F., Wang, A., White, A., Young, A., Yu, R., Aspuru-Guzik, A.: SELFIES and the future of molecular string representations. Patterns 3

Original

2023

Earlier work this paper cites.

Heller, S.R., McNaught, A., Pletnev, I., Stein, S., Tchekhovskoi, D.: InChI, the IUPAC International Chemical Identifier. Journal of Cheminformatics 7

2023

Earlier work this paper cites.

Brammer, J.C., Blanke, G., Kellner, C., Hoffmann, A., Herres-Pawlis, S., Schatzschneider, U.: TUCAN: A molecular identifier and descriptor applicable to the whole periodic table from hydrogen to oganesson. Journal of Cheminformatics 14

2023

Earlier work this paper cites.

Restrepo, G.: Chemical space: limits, evolution and modelling of an object bigger than our universal library. Digital Discovery 1

2023

Cited alongside, same era.

Kusner, M.J., Paige, B., Hernández-Lobato, J.M.: Grammar Variational Autoencoder. arXiv. arXiv:1703.01925 [stat] (2017). https://doi.org/10.48550/arXiv.1703.01925 . http://arxiv.org/abs/1703.01925 Accessed 2023-07-27

Original

2023

Cited alongside, same era.

Gómez-Bombarelli, R., Wei, J.N., Duvenaud, D., Hernández-Lobato, J.M., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T.D., Adams, R.P., Aspuru-Guzik, A.: Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. ACS Central Science 4

2023

Cited alongside, same era.

Öztürk, H., Özgür, A., Schwaller, P., Laino, T., Ozkirimli, E.: Exploring chemical space using natural language processing methodologies for drug discovery. Drug Discovery Today 25

2023

Cited alongside, same era.

Christofidellis, D., Giannone, G., Born, J., Winther, O., Laino, T., Manica, M.: Unifying Molecular and Textual Representations via Multi-task Language Modelling. arXiv. arXiv:2301.12586 [cs] (2023). https://doi.org/10.48550/arXiv.2301.12586 . http://arxiv.org/abs/2301.12586 Accessed 2023-07-27

Original

2023

Closest in time.

Alberts, M., Laino, T., Vaucher, A.C.: Leveraging Infrared Spectroscopy for Automated Structure Elucidation. preprint, Chemistry (May 2023). https://doi.org/10.26434/chemrxiv-2023-5v27f . https://chemrxiv.org/engage/chemrxiv/article-details/645df5cbf2112b41e96da616 Accessed 2023-07-25

2023

Closest in time.

Raschka, S.: Finetuning Large Language Models (2023). https://magazine.sebastianraschka.com/p/finetuning-large-language-models?utm_campaign=post Accessed 2023-05-17

2023

Closest in time.

Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., Qiao, Y.: LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arXiv. arXiv:2303.16199 [cs] (2023). http://arxiv.org/abs/2303.16199 Accessed 2023-05-05

Original