2024

A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?

Kristiadi, Agustinus, Strieth-Kalthoff, Felix, Skreta, Marta et al.

Understand

Automation is one of the cornerstones of contemporary material discovery.

Bayesian optimization (BO) is an essential part of such workflows, enabling scientists to leverage prior domain knowledge into efficient exploration of a large molecular space.
While such prior knowledge can take many forms, there has been significant fanfare around the ancillary scientific knowledge encapsulated in large language models (LLMs).
However, existing work thus far has only explored LLMs for heuristic materials searches.

Built on

On the likelihood that one unknown probability exceeds another in view of the evidence of two samples
Thompson, W. R · 1933
Earlier work this paper cites.
The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service
Morgan, H. L · 1965
Earlier work this paper cites.
On Bayesian methods for seeking the extremum
Močkus, J · 1975
Earlier work this paper cites.
Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules
Weininger, D · 1988
Earlier work this paper cites.
Efficient global optimization of expensive black-box functions
Jones, D. R., Schonlau, M., and Welch, W. J · 1998
Earlier work this paper cites.
Evolutionary algorithms for multiobjective optimization: Methods and applications , volume 63
Zitzler, E · 1999
Earlier work this paper cites.
Finite-time analysis of the multi-armed bandit problem
Auer, P., Cesa-Bianchi, N., and Fischer, P · 2002
Earlier work this paper cites.
Virtual screening of chemical libraries
Shoichet, B. K · 2004
Earlier work this paper cites.
Gaussian processes in machine learning
Rasmussen, C. E. and Williams, C. K. I · 2006
Earlier work this paper cites.
Virtual screening: An endless staircase?
Schneider, G · 2010
Earlier work this paper cites.
Practical Bayesian optimization of machine learning algorithms
Snoek, J., Larochelle, H., and Adams, R. P · 2012
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
What is high-throughput virtual screening? A perspective from organic materials discovery
Pyzer-Knapp, E. O., Suh, C., Gómez-Bombarelli, R., Aguilera-Iparraguirre, J., and Aspuru-Guzik, A · 2015
Earlier work this paper cites.
The harvard organic photovoltaic dataset
Lopez, S. A., Pyzer-Knapp, E. O., Simm, G. N., Lutzow, T., Li, K., Seress, L. R., Hachmann, J., and Aspuru-Guzik, A · 2016
Earlier work this paper cites.
Deep kernel learning
Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P · 2016
Earlier work this paper cites.
Practical Gauss-Newton optimisation for deep learning
Botev, A., Ritter, H., and Barber, D · 2017
Earlier work this paper cites.
Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space
Hernández-Lobato, J. M., Requeima, J., Pyzer-Knapp, E. O., and Aspuru-Guzik, A · 2017
Earlier work this paper cites.
SGDR: Stochastic gradient descent with warm restarts
Loshchilov, I. and Hutter, F · 2017
Earlier work this paper cites.
Attention is all you need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017
Earlier work this paper cites.
Automatic chemical design using a data-driven continuous representation of molecules
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., and Aspuru-Guzik, A · 2018
Earlier work this paper cites.
Neural tangent kernel: Convergence and generalization in neural networks
Jacot, A., Gabriel, F., and Hongler, C · 2018
Earlier work this paper cites.
A scalable Laplace approximation for neural networks
Ritter, H., Botev, A., and Barber, D · 2018
Earlier work this paper cites.
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S · 2019
Earlier work this paper cites.
Ultra-large library docking for discovering new chemotypes
Lyu, J., Wang, S., Balius, T. E., Singh, I., Levit, A., Moroz, Y. S., O’Meara, M. J., Che, T., Algaa, E., Tolmachova, K., Tolmachev, A. A., Shoichet, B. K., Roth, B. L., and Irwin, J. J · 2019
Earlier work this paper cites.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al · 2019
Earlier work this paper cites.
Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence
Sun, C., Huang, L., and Qiu, X · 2019
Earlier work this paper cites.
Huggingface’s transformers: State-of-the-art natural language processing
Original
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al · 2019
Earlier work this paper cites.
Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning
Zhang, Y. and Lee, A. A · 2019
Earlier work this paper cites.

Similar

BoTorch: A framework for efficient Monte-Carlo Bayesian optimization
Balandat, M., Karrer, B., Jiang, D., Daulton, S., Letham, B., Wilson, A. G., and Bakshy, E · 2020
Cited alongside, same era.
Language models are few-shot learners
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al · 2020
Cited alongside, same era.
LEGAL-BERT: The muppets straight out of law school
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I · 2020
Cited alongside, same era.
ChemBERTa: Large-scale self-supervised pretraining for molecular property prediction
Original
Chithrananda, S., Grand, G., and Ramsundar, B · 2020
Cited alongside, same era.
Understanding, values and the aims of science
de Regt, H. W · 2020
Cited alongside, same era.
BINOCULARS for efficient, nonmyopic sequential experimental design
Jiang, S., Chai, H., Gonzalez, J., and Garnett, R · 2020
Cited alongside, same era.
ChemBO: Bayesian optimization of small organic molecules with synthesizable recommendations
Korovina, K., Xu, S., Kandasamy, K., Neiswanger, W., Poczos, B., Schneider, J., and Xing, E. P · 2020
Cited alongside, same era.
A flexible framework for multi-objective Bayesian optimization using random scalarizations
Paria, B., Kandasamy, K., and Póczos, B · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J · 2020
Cited alongside, same era.
The cost of training NLP models: A concise overview
Original
Sharir, O., Peleg, B., and Shoham, Y · 2020
Cited alongside, same era.
Sample-efficient optimization in the latent space of deep generative models via weighted retraining
Tripp, A., Daxberger, E., and Hernández-Lobato, J. M · 2020
Cited alongside, same era.
Discovery of energy storage molecular materials using quantum chemistry-guided multiobjective bayesian optimization
Agarwal, G., Doan, H. A., Robertson, L. A., Zhang, L., and Assary, R. S · 2021
Cited alongside, same era.

Then

Chemical space: limits, evolution and modelling of an object bigger than our universal library
Restrepo, G · 2022
Later among the works it cites.
Large-scale chemical language representations capture molecular structure and properties
Ross, J., Belgodere, B., Chenthamarakshan, V., Padhi, I., Mroueh, Y., and Das, P · 2022
Later among the works it cites.
Pre-train your loss: Easy Bayesian transfer learning with informative priors
Shwartz-Ziv, R., Goldblum, M., Souri, H., Kapoor, S., Zhu, C., LeCun, Y., and Wilson, A. G · 2022
Later among the works it cites.
Accelerating Bayesian optimization for biological sequence design with denoising autoencoders
Stanton, S., Maddox, W., Gruver, N., Maffettone, P., Delaney, E., Greenside, P., and Wilson, A. G · 2022
Later among the works it cites.
Closed-loop transfer enables AI to yield chemical knowledge
Angello, N., Friday, D., Hwang, C., Yi, S., Cheng, A., Torres-Flores, T., Jira, E., Wang, W., Aspuru-Guzik, A., Burke, M., and et al · 2023
Later among the works it cites.
Riemannian Laplace approximations for Bayesian neural networks
Bergamin, F., Moreno-Muñoz, P., Hauberg, S., and Arvanitidis, G · 2023
Later among the works it cites.
Unifying molecular and textual representations via multi-task language modelling
Christofidellis, D., Giannone, G., Born, J., Winther, O., Laino, T., and Manica, M · 2023
Later among the works it cites.
The Enamine REAL database, 2023
Enamine Ltd · 2023
Later among the works it cites.
Bayesian optimization
Garnett, R · 2023
Later among the works it cites.
VirtualFlow 2.0 - the next generation drug discovery platform enabling adaptive screens of 69 billion molecules
Gorgulla, C., Nigam, A., Koop, M., Çınaroğlu, S. S., Secker, C., Haddadnia, M., Kumar, A., Malets, Y., Hasson, A., Li, M., Tang, M., Levin-Konigsberg, R., Radchenko, D., Kumar, A., Gehev, M., Aquilanti, P.-Y., Gabb, H., Alhossary, A., Wagner, G., Aspuru-Guzik, A., Moroz, Y. S., Fackeldey, K., and Arthanari, H · 2023
Later among the works it cites.
From alchemist to AI chemist
Greenaway, R. L., Jelfs, K. E., Spivey, A. C., and Yaliraki, S. N · 2023
Later among the works it cites.
GAUCHE: A library for Gaussian processes in chemistry
Griffiths, R.-R., Klarner, L., Moss, H. B., Ravuri, A., Truong, S., Stanton, S., Tom, G., Rankovic, B., Du, Y., Jamasb, A., Deshwal, A., Schwartz, J., Tripp, A., Kell, G., Frieder, S., Bourached, A., Chan, A., Moss, J., Guo, C., Durholt, J., Chaurasia, S., Strieth-Kalthoff, F., Lee, A. A., Cheng, B., Aspuru-Guzik, A., Schwaller, P., and Tang, J · 2023
Later among the works it cites.
Large language models are zero-shot time series forecasters
Gruver, N., Finzi, M., Qiu, S., and Wilson, A. G · 2023
Later among the works it cites.
What can large language models do in chemistry? A comprehensive benchmark on eight tasks
Original
Guo, T., Guo, K., Nan, B., Liang, Z., Guo, Z., Chawla, N. V., Wiest, O., and Zhang, X · 2023
Later among the works it cites.
In-context learning of large language models explained as kernel regression
Original
Han, C., Wang, Z., Zhao, H., and Ji, H · 2023
Later among the works it cites.
Olympus, enhanced: Benchmarking mixed-parameter and multi-objective optimization in chemistry and materials science
Hickman, R., Parakh, P., Cheng, A., Ai, Q., Schrier, J., Aldeghi, M., and Aspuru-Guzik, A · 2023
Later among the works it cites.
ChatGPT for good? on opportunities and challenges of large language models for education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al · 2023
Later among the works it cites.
Promises and pitfalls of the linearized Laplace in Bayesian optimization
Kristiadi, A., Immer, A., Eschenhagen, R., and Fortuin, V · 2023
Later among the works it cites.
The impact of large language models on scientific discovery: a preliminary study using GPT-4
Original
Microsoft Research AI4Science and Microsoft Azure Quantum · 2023
Later among the works it cites.
GPT-4 technical report
Original
OpenAI · 2023
Later among the works it cites.
Bayesian optimization of catalysts with in-context learning
Original
Ramos, M. C., Michtavy, S. S., Porosoff, M. D., and White, A. D · 2023
Later among the works it cites.
BoChemian: Large language model embeddings for Bayesian optimization of chemical reactions
Ranković, B. and Schwaller, P · 2023
Later among the works it cites.
Scientific discovery in the age of artificial intelligence
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Katwyk, P. V., Deac, A., Anandkumar, A., Bergen, K., Gomes, C. P., Ho, S., Kohli, P., Lasenby, J., Leskovec, J., Liu, T.-Y., Manrai, A., Marks, D., Ramsundar, B., Song, L., Sun, J., Tang, J., Velickovic, P., Welling, M., Zhang, L., Coley, C. W., Bengio, Y., and Zitnik, M · 2023
Later among the works it cites.
Bayesian low-rank adaptation for large language models
Original
Yang, A. X., Robeyns, M., Wang, X., and Aitchison, L · 2023
Later among the works it cites.
A study of Bayesian neural network surrogates for Bayesian optimization
Li, Y. L., Rudner, T. G., and Wilson, A. G · 2024
Closest in time.
Large language models to enhance Bayesian optimization
Liu, T., Astorga, N., Seedat, N., and van der Schaar, M · 2024
Closest in time.
Delocalized, asynchronous, closed-loop discovery of organic laser emitters
Strieth-Kalthoff, F., Hao, H., Rathore, V., Derasp, J., Gaudin, T., Angello, N. H., Seifrid, M., Trushina, E., Guy, M., Liu, J., and et al · 2024
Closest in time.
Self-driving laboratories for chemistry and materials sciences
Tom, G., Schmid, S. P., Baird, S. G., Cao, Y., Darvish, K., Hao, H., Lo, S., Pablo-Garcia, S., Rajaonson, E. M., Skreta, M., Yoshikawa, N., Corapi, S., Akkoc, G. D., Strieth-Kalthoff, F., Seifrid, M., and Aspuru-Guzik, A · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…