Fetching the paper…

Reading the bibliography…

2024

Materials science in the era of large language models: a perspective

Lei, Ge, Docherty, Ronan, Cooper, Samuel J.

Original
PDF
Post on X

Understand

Large Language Models (LLMs) have garnered considerable interest due to their impressive natural language capabilities, which in conjunction with various emergent properties make them versatile tools in workflows ranging from complex code generation to heuristic finding for combinatorial problems.

In this paper we offer a perspective on their applicability to materials science research, arguing their ability to handle ambiguous requirements across a range of tasks and disciplines mean they could be a powerful tool to aid researchers.
We qualitatively examine basic LLM theory, connecting it to relevant properties and techniques in the literature before providing two case studies that demonstrate their use in task automation and knowledge extraction at-scale.
At their current stage of development, we argue LLMs should be viewed less as oracles of novel insight, and more as tireless workers that can accelerate and unify exploration across domains.

Materials science in the era of large language models: a perspective · Around

Built on

D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A Learning Algorithm for Boltzmann Machines*,” Cognitive Science
1985
Earlier work this paper cites.
https://www.doitpoms.ac.uk/index.php
J. Eliot, “Doitpoms micrograph library,” 2000 · 2000
Earlier work this paper cites.
D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473
Original
2016
Earlier work this paper cites.
M. H. Rafiei, W. H. Khushefati, R. Demirboga, and H. Adeli, “Neural network, machine learning, and evolutionary approaches for concrete material characterization,” ACI Materials Journal
2016
Earlier work this paper cites.
C. Clark and S. Divvala, “PDFFigures 2.0: Mining figures from research papers,” in 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL)
2016
Earlier work this paper cites.
C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting unreasonable effectiveness of data in deep learning era,” arXiv preprint arXiv:1707.02968
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems
2017
Earlier work this paper cites.
J. Ficler and Y. Goldberg, “Controlling linguistic style aspects in neural language generation,” arXiv preprint arXiv:21707.02633
2017
Earlier work this paper cites.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347
Original
2017
Earlier work this paper cites.
E. Williams, J. Moore, S. W. Li, G. Rustici, A. Tarkowska, A. Chessel, S. Leo, B. Antal, R. K. Ferguson, U. Sarkans, A. Brazma, R. E. Carazo Salas, and J. R. Swedlow, “Image Data Resource: a bioimage data integration and publication platform,” Nature Methods
2017
Earlier work this paper cites.
M. Freitag and Y. Al-Onaizan, “Beam Search Strategies for Neural Machine Translation,” in Proceedings of the First Workshop on Neural Machine Translation
2017
Earlier work this paper cites.
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pre-training,” 2018
2018
Earlier work this paper cites.
R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Sánchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik, “Automatic chemical design using a data-driven continuous representation of molecules,” ACS central science
2018
Earlier work this paper cites.
R. Aversa, M. H. Modarres, S. Cozzini, R. Ciancio, and A. Chiusole, “The first annotated set of scanning electron microscopy images for nanoscience,” Scientific Data
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805
Original
2019
Earlier work this paper cites.
F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?,” arXiv preprint arXiv:1909.01066
Original
2019
Earlier work this paper cites.
I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,” arXiv preprint arXiv:1903.10676
Original
2019
Earlier work this paper cites.
C. Sun, A. Myers, C. Vondrick, K. Murphy, and C. Schmid, “Videobert: A joint model for video and language representation learning,” arXiv preprint arXiv:1904.01766
Original
2019
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” arXiv preprint arXiv:2005.14165
Original
2020
Earlier work this paper cites.
I. Beltagy, M. E. Peters, and A. Cohan, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150
Original
2020
Earlier work this paper cites.
R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence
2020
Earlier work this paper cites.
A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” 2020
2020
Earlier work this paper cites.
Q. Zhou, S. Lu, Y. Wu, and J. Wang, “Property-oriented material design based on a data-driven machine learning technique,” The journal of physical chemistry letters
2020
Earlier work this paper cites.
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” arXiv preprint arXiv:2101.00027
Original
2020
Earlier work this paper cites.
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, and T. Rocktäschel, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” Advances in Neural Information Processing Systems
2020
Earlier work this paper cites.
A. Coja-Oghlan, P. Loick, B. F. Mezei, and G. B. Sorkin, “The ising antiferromagnet and max cut on random regular graphs,” arXiv preprint arXiv:2009.10483
Original
2020
Earlier work this paper cites.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, and J. Clark, “Learning transferable visual models from natural language supervision,” in International conference on machine learning
2021
Earlier work this paper cites.
P. Dufter, M. Schmitt, and H. Schütze, “Position information in transformers: An overview,” arXiv preprint arXiv:2102.11090
Original
2021
Earlier work this paper cites.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging Properties in Self-Supervised Vision Transformers,” arXiv preprint arXiv:2104.14294
Original
2021
Earlier work this paper cites.
D. Chen, K. Gao, D. D. Nguyen, X. Chen, Y. Jiang, G.-W. Wei, and F. Pan, “Algebraic graph-assisted bidirectional transformers for molecular property prediction,” Nature communications
2021
Earlier work this paper cites.
F. Kuniyoshi, J. Ozawa, and M. Miwa, “Analyzing research trends in inorganic materials literature using nlp,” in Joint European Conference on Machine Learning and Knowledge Discovery in Databases
2021
Earlier work this paper cites.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685
Original
2021
Earlier work this paper cites.
B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” arXiv preprint arXiv:2104.08691
Original
2021
Earlier work this paper cites.
X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190
Original
2021
Earlier work this paper cites.
R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, and W. Saunders, “Webgpt: Browser-assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332
Original
2021
Earlier work this paper cites.
S. Kench and S. J. Cooper, “Generating three-dimensional structures from a two-dimensional slice with generative adversarial network-based dimensionality expansion,” Nature Machine Intelligence
2021
Earlier work this paper cites.

Similar

J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus, “Emergent Abilities of Large Language Models,” arXiv preprint arXiv:2206.07682
Original
2022
Cited alongside, same era.
https://huggingface.co/blog/rlhf
N. Lambert, L. Castricato, L. von Werra, and A. Havrilla, “Illustrating reinforcement learning from human feedback (rlhf),” 2022 · 2022
Cited alongside, same era.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” arXiv preprint arXiv: 2203.02155
2022
Cited alongside, same era.
D. P. Finegan, I. Squires, A. Dahari, S. Kench, K. L. Jungjohann, and S. J. Cooper, “Machine-learning-driven advanced characterization of battery electrodes,” ACS Energy Letters
2022
Cited alongside, same era.

Then

A. N. Rubungo, C. Arnold, B. P. Rand, and A. B. Dieng, “Llm-prop: Predicting physical and electronic properties of crystalline solids from their text descriptions,” arXiv preprint arXiv:2310.14029
Original
2023
Later among the works it cites.
S. J. Yang, S. Li, S. Venugopalan, V. Tshitoyan, M. Aykol, A. Merchant, E. D. Cubuk, and G. Cheon, “Accurate prediction of experimental band gaps from large language model-based data extraction,” arXiv preprint arXiv:2311.13778
Original
2023
Later among the works it cites.
M. R. AI4Science and M. A. Quantum, “The impact of large language models on scientific discovery: a preliminary study using gpt-4,” arXiv preprint arXiv:2311.07361
Original
2023
Later among the works it cites.
N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” arXiv preprint arXiv:2303.11366
Original
2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

E. Champa-Bujaico, P. García-Díaz, and A. M. Díez-Pascual, “Machine learning for property prediction and optimization of polymeric nanocomposites: a state-of-the-art,” International Journal of Molecular Sciences

2022

Cited alongside, same era.

A. Goetz, A. R. Durmaz, M. Müller, A. Thomas, D. Britz, P. Kerfriden, and C. Eberl, “Addressing materials’ microstructure diversity using transfer learning,” npj Computational Materials

2022

Cited alongside, same era.

S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?,” arXiv preprint arXiv:2202.12837

2022

Cited alongside, same era.

J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems

2022

Cited alongside, same era.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629

2022

Cited alongside, same era.

J. Huang, S. S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han, “Large language models can self-improve,” arXiv preprint arXiv:2210.11610

2022

Cited alongside, same era.

T. Gupta, M. Zaki, N. A. Krishnan, and Mausam, “Matscibert: A materials domain language model for text mining and information extraction,” npj Computational Materials

2022

Cited alongside, same era.

J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” arXiv preprint arXiv:2201.12086

2022

Cited alongside, same era.

V. Nair, E. Schumacher, G. Tso, and A. Kannan, “Dera: enhancing large language model completions with dialog-enabled resolving agents,” arXiv preprint arXiv:2303.17071

2023

Later among the works it cites.

Y. Lin, L. Tan, H. Lin, Z. Zheng, R. Pi, J. Zhang, S. Diao, H. Wang, H. Zhao, and Y. Yao, “Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models,” arXiv preprint arXiv:2309.06256

2023

Later among the works it cites.

Z. Hu, Y. Lan, L. Wang, W. Xu, E.-P. Lim, R. K.-W. Lee, L. Bing, and S. Poria, “Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models,” arXiv preprint arXiv:2304.01933

2023

Later among the works it cites.

L. Moussiades and G. Zografos, “Openai’s gpt4 as coding assistant,” arXiv preprint arXiv:2309.12732

2023

Later among the works it cites.

P. Xu, X. Zhu, and D. A. Clifton, “Multimodal learning with transformers: A survey,” arXiv preprint arXiv:2206.06488

2023

Later among the works it cites.

H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” 2023

2023

Later among the works it cites.

J. Lin, R. Pradeep, T. Teofili, and J. Xian, “Vector search with openai embeddings: Lucene is all you need,” arXiv preprint arXiv:2308.14963

2023

Later among the works it cites.

T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” arXiv preprint arXiv:2302.04761

2023

Later among the works it cites.

C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan, “Visual chatgpt: Talking, drawing and editing with visual foundation models,” arXiv preprint arXiv:2303.04671

2023

Later among the works it cites.

D. Surís, S. Menon, and C. Vondrick, “Vipergpt: Visual inference via python execution for reasoning,” arXiv preprint arXiv:2303.08128

2023

Later among the works it cites.

T. Cai, X. Wang, T. Ma, X. Chen, and D. Zhou, “Large language models as tool makers,” arXiv preprint arXiv:2305.17126

2023

Later among the works it cites.

J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology

2023

Later among the works it cites.

C. Packer, V. Fang, S. G. Patil, K. Lin, S. Wooders, and J. E. Gonzalez, “Memgpt: Towards llms as operating systems,” arXiv preprint arXiv:2310.08560

2023

Later among the works it cites.

Y. Hu, Q. Xie, V. Jain, J. Francis, J. Patrikar, N. Keetha, S. Kim, Y. Xie, T. Zhang, S. Zhao, Y. Q. Chong, C. Wang, K. Sycara, M. Johnson-Roberson, D. Batra, X. Wang, S. Scherer, Z. Kira, F. Xia, and Y. Bisk, “Toward general-purpose robots via foundation models: A survey and meta-analysis,” arXiv preprint arXiv:2312.08782

2023

Later among the works it cites.

D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence, “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378

Original

2023

Later among the works it cites.

S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang, “Language-driven representation learning for robotics,” arXiv preprint arXiv:2302.12766

2023

Later among the works it cites.

Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” arXiv preprint arXiv:2310.12931

2023

Later among the works it cites.

S. Kench, I. Squires, and S. Cooper, “Taufactor 2: A gpu accelerated python tool for microstructural analysis,” Journal of Open Source Software

2023

Later among the works it cites.

L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” arXiv preprint arXiv:2311.05232

2023

Later among the works it cites.

Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” 2024

2024

Closest in time.

D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kinney, O. Tafjord, A. H. Jha, H. Ivison, I. Magnusson, Y. Wang, S. Arora, D. Atkinson, R. Authur, K. R. Chandu, A. Cohan, J. Dumas, Y. Elazar, Y. Gu, J. Hessel, T. Khot, W. Merrill, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, V. Pyatkin, A. Ravichander, D. Schwenk, S. Shah, W. Smith, E. Strubell, N. Subramani, M. Wortsman, P. Dasigi, N. Lambert, K. Richardson, L. Zettlemoyer, J. Dodge, K. Lo, L. Soldaini, N. A. Smith, and H. Hajishirzi, “OLMo: Accelerating the Science of Language Models,” arXiv preprint arXiv: 2402.00838

2024

Closest in time.

B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi, “Mathematical discoveries from program search with large language models,” Nature

2024

Closest in time.

C. Zeni, R. Pinsler, D. Zügner, A. Fowler, M. Horton, X. Fu, S. Shysheya, J. Crabbé, L. Sun, J. Smith, B. Nguyen, H. Schulz, S. Lewis, C.-W. Huang, Z. Lu, Y. Zhou, H. Yang, H. Hao, J. Li, R. Tomioka, and T. Xie, “Mattergen: a generative model for inorganic materials design,” preprint arXiv:2312.03687

Original

2024

Closest in time.

S. M. Bsharat, A. Myrzakhan, and Z. Shen, “Principled instructions are all you need for questioning llama-1/2, gpt-3.5/4,” arXiv preprint arXiv:2312.16171

2024

Closest in time.

G. Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” 2024

2024

Closest in time.

OpenAI, “Sora: Creating video from text.” https://openai.com/sora , 2024

2024

Closest in time.

Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, Q. Guo, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997

2024

Closest in time.

Publisher: American Chemical Society

M. J. Buehler, “Generative Retrieval-Augmented Ontologic Graph and Multiagent Strategies for Interpretive Large Language Model-Based Materials Design,” ACS Engineering Au · 2024

Closest in time.

E. Davis, “Using a large language model to generate program mutations for a genetic algorithm to search for solutions to combinatorial problems: Review of (Romera-Paredes et al., 2023).,” Jan. 2024

2024

Closest in time.

T. Ridnik, D. Kredo, and I. Friedman, “Code generation with alphacodium: From prompt engineering to flow engineering,” arXiv preprint arXiv:2401.08500

2024

Closest in time.

Publisher: RSC

Z. Zheng, Z. He, O. Khattab, N. Rampal, M. A. Zaharia, C. Borgs, J. T. Chayes, and O. M. Yaghi, “Image and data mining in reticular chemistry powered by GPT-4V,” Digital Discovery · 2024

Closest in time.