AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., in Advances in Neural Information Processing Systems , Vol. 33 (Curran Associates, Inc., 2020) pp. 1877–1901

1901

Earlier work this paper cites.

I. Turc, M.-W. Chang, K. Lee and K. Toutanova, “Well-Read Students Learn Better: On the Importance of Pre-training Compact Models,” (2019), arXiv:1908.08962

Original

1908

Earlier work this paper cites.

T. Schick and H. Schütze, “It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners,” (2021), arXiv:2009.07118

Original

2009

Earlier work this paper cites.

J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher,” (2021), arXiv:2112.11446v2

Original

2021

Earlier work this paper cites.

“Megatron-DeepSpeed,” (2021), https://github.com/deepspeedai/Megatron-DeepSpeed

2021

Earlier work this paper cites.

D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song and J. Steinhardt, “Measuring Mathematical Problem Solving With the MATH Dataset,” (2021), arXiv:2103.03874

Original

2021

Earlier work this paper cites.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark et al., “Training Compute-Optimal Large Language Models,” (2022), arXiv:2203.15556

Original

2022

Earlier work this paper cites.

“tiktoken,” (2022), https://github.com/openai/tiktoken

2022

Earlier work this paper cites.

S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti et al., “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model,” (2022), arXiv:2201.11990

Original

2022

Earlier work this paper cites.

M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou and J. Wei, “Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them,” (2022), arXiv:2210.09261

Original

2022

Earlier work this paper cites.

T. D. Nguyen, Y.-S. Ting, I. Ciucă, C. O’Neill, Z.-C. Sun, M. Jabłońska, S. Kruk, E. Perkowski, J. Miller, J. Li et al., “AstroLLaMA: Towards Specialized Foundation Models in Astronomy,” (2023), arXiv:2309.06126v1

Original

2023

Earlier work this paper cites.

G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei and J. Launay, “The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only,” (2023), arXiv:2306.01116v1

Original

2023

Cited alongside, same era.

L. Blecher, G. Cucurull, T. Scialom and R. Stojnic, “Nougat: Neural Optical Understanding for Academic Documents,” (2023), arXiv:2308.13418

Original

2023

Cited alongside, same era.

S. Dash, I. Lyngaas, J. Yin, X. Wang, R. Egele, G. Cong, F. Wang and P. Balaprakash, “Optimizing Distributed Training on Frontier for Large Language Models,” (2023), arXiv:2312.12705v2

Original

2023

Cited alongside, same era.

A. Dassanaike-Perera, S. Waiwitlikhit and K. Gilbai, Cuts and Stitches: Does Model Merging Produce Better Multitask Learners? , Stanford CS224N Default Project (Stanford, 2023)

2023

Cited alongside, same era.

Z. Sun, Y.-S. Ting, Y. Liang, N. Duan, S. Huang and Z. Cai, “Interpreting Multi-band Galaxy Observations with Large Language Model-Based Agents,” (2024), arXiv:2409.14807

Original

2024

Closest in time.

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The Llama 3 Herd of Models,” (2024), arXiv:2407.21783

Original

2024

Closest in time.

Y.-S. Ting, T. D. Nguyen, T. Ghosal, R. Pan, H. Arora, Z. Sun, T. de Haan, N. Ramachandra, A. Wells, S. Madireddy and A. Accomazzi, “AstroMLab 1: Who Wins Astronomy Jeopardy!?” (2024), arXiv:2407.11194v1

Original

2024

Closest in time.

R. Li, Y. Wei, M. Zhang, N. Yu, H. Hu and H. Peng, “ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws,” (2024), arXiv:2408.08310v1

Original

2024

Closest in time.