How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020 · 1901

Earlier work this paper cites.

Transformer-xl: Attentive language models beyond a fixed-length context

Original

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019 · 1901

Earlier work this paper cites.

On the turing completeness of modern neural network architectures

Original

Jorge Pérez, Javier Marinković, and Pablo Barceló. 2019 · 1901

Earlier work this paper cites.

Are transformers universal approximators of sequence-to-sequence functions?

Original

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. 2019 · 1912

Earlier work this paper cites.

Lower bounds for the size of circuits of bounded depth with basis fˆ; g

Alexander A Razborov. 1987 · 1987

Earlier work this paper cites.

Algebraic methods in the theory of lower bounds for boolean circuit complexity

Roman Smolensky. 1987 · 1987

Earlier work this paper cites.

Ieee standard 754 for binary floating-point arithmetic

William Kahan. 1996 · 1996

Earlier work this paper cites.

Latent dirichlet allocation

David M Blei, Andrew Y Ng, and Michael I Jordan. 2003 · 2003

Earlier work this paper cites.

Ieee standard for floating-point arithmetic

IEEE. 2019 · 2008

Earlier work this paper cites.

Computational complexity: a modern approach

Sanjeev Arora and Boaz Barak. 2009 · 2009

Earlier work this paper cites.

On the ability and limitations of transformers to recognize formal languages

Original

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020 · 2009

Earlier work this paper cites.

Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding

Original

Song Han, Huizi Mao, and William J Dally. 2015 · 2015

Earlier work this paper cites.

Gaussian error linear units (gelus)

Original

Dan Hendrycks and Kevin Gimpel. 2016 · 2016

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

Analysing mathematical reasoning abilities of neural models

David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019 · 2019

Earlier work this paper cites.

Theoretical limitations of self-attention in neural sequence models

Michael Hahn. 2020 · 2020

Earlier work this paper cites.

O (n) connections are expressive enough: Universal approximability of sparse transformers

Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. 2020 · 2020

Earlier work this paper cites.

A mathematical framework for transformer circuits

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Investigating the limitations of transformers with simple arithmetic tasks

Original

Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2021 · 2021

Earlier work this paper cites.

Attention is turing complete

Jorge Pérez, Pablo Barceló, and Javier Marinkovic. 2021 · 2021

Earlier work this paper cites.

Thinking like transformers

Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021 · 2021

Earlier work this paper cites.

Self-attention networks can process bounded hierarchical languages

Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. 2021 · 2021

Earlier work this paper cites.

What learning algorithm is in-context learning? investigations with linear models

Original

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022 · 2022

Earlier work this paper cites.

What can transformers learn in-context? a case study of simple function classes

Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. 2022 · 2022

Earlier work this paper cites.

Formal language recognition by hard attention transformers: Perspectives from circuit complexity

Yiding Hao, Dana Angluin, and Robert Frank. 2022 · 2022

Earlier work this paper cites.

Your transformer may not be as powerful as you expect

Shengjie Luo, Shanda Li, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, and Di He. 2022 · 2022

Earlier work this paper cites.

Saturated transformers are constant-depth threshold circuits

William Merrill, Ashish Sabharwal, and Noah A Smith. 2022 · 2022

Earlier work this paper cites.

LILA: A unified benchmark for mathematical reasoning

Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022 · 2022

Cited alongside, same era.

In-context learning and induction heads

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022 · 2022

Cited alongside, same era.

Sumformer: Universal approximation for efficient transformers

Silas Alberti, Niclas Dern, Laura Thesing, and Gitta Kutyniok. 2023 · 2023

Cited alongside, same era.

Text summarization using large language models: A comparative study of mpt-7b-instruct, falcon-7b-instruct, and openai chat-gpt models

Original

Lochan Basyal and Mihir Sanghvi. 2023 · 2023

Cited alongside, same era.

Sparks of artificial general intelligence: Early experiments with gpt-4

The claude 3 model family: Opus, sonnet, haiku

Anthropic. 2024 · 2024

Closest in time.

Large language monkeys: Scaling inference compute with repeated sampling

Original

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. 2024 · 2024

Closest in time.

Qlora: Efficient finetuning of quantized llms

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024 · 2024

Closest in time.

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024

Closest in time.

Mathematical capabilities of chatgpt

Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. 2024 · 2024

Closest in time.