Language models are better than humans at next-token prediction

A comparison of human and statistical language model performance using missing-word tests

M. Owens, P. O’Boyle, J. McMahon, J. Ming, and F. J. Smith · 1997

Earlier work this paper cites.

One billion word benchmark for measuring progress in statistical language modeling

Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, T. Brants, Phillip Todd Koehn, and Tony Robinson · 2014

Earlier work this paper cites.

Building a large annotated corpus of english: The penn treebank

Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini · 2017

Earlier work this paper cites.

Estimation of gap between current language models and human performance

Xiaoyu Shen, Youssef Oualil, Clayton Greenberg, Mittul Singh, and Dietrich Klakow · 2017

Earlier work this paper cites.

Openwebtext corpus

A. Gokaslan and V. Cohen · 2019

Earlier work this paper cites.

Chris olah’s views on agi safety, 2019

E Hurbinger · 2019

Cited alongside, same era.

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020

Cited alongside, same era.

The pile: An 800gb dataset of diverse text for language modeling

Original

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al · 2020

Cited alongside, same era.

Openai’s gpt-3 language model, 2020

Steve Omohundro · 2020

Cited alongside, same era.

Better language models and their implications, 2019a

Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, and Ilya Sutskever

Cited in the paper.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al

Cited in the paper.

GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman · 2021

Later among the works it cites.

Measuring mathematical problem solving with the math dataset

Original

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xiaodong Song, and Jacob Steinhardt · 2021

Later among the works it cites.

GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model

Ben Wang and Aran Komatsuzaki · 2021

Later among the works it cites.

Shared computational principles for language processing in humans and deep language models

Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, and Uri Hasson · 2022

Closest in time.