Built on

Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720
Original
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington · 1902
Earlier work this paper cites.
Generating long sequences with sparse transformers
Original
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever · 1904
Earlier work this paper cites.
Analysing mathematical reasoning abilities of neural models
Original
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli · 1904
Earlier work this paper cites.
One epoch is all you need, 2019, arXiv:1906.06669
Original
Aran Komatsuzaki · 1906
Earlier work this paper cites.
Scaling autoregressive video models, 2019, 1906.02634
Original
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit · 1906
Earlier work this paper cites.
Compositionality decomposed: how do neural networks generalise?, 2019, 1908.08351
Original
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni · 1908
Earlier work this paper cites.
A constructive prediction of the generalization error across scales, 2019, 1909.12673
Original
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit · 1909
Earlier work this paper cites.
Enhancing the transformer with explicit relational encoding for math problem solving, 2019, 1910.06611
Original
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao · 1910
Earlier work this paper cites.
Scaling laws for neural language models, 2020, 2001.08361
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2001
Earlier work this paper cites.
Train large, then compress: Rethinking model size for efficient training and inference of transformers, 2020, 2002.11794
Original
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez · 2002
Earlier work this paper cites.

Similar

The large learning rate phase of deep learning: the catapult mechanism, 2020, 2003.02218
Original
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari · 2003
Cited alongside, same era.
Recipes for building an open-domain chatbot, 2020, 2004.13637
Original
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston · 2004
Cited alongside, same era.
A neural scaling law from the dimension of the data manifold, 2020, 2004.10802
Original
Utkarsh Sharma and Jared Kaplan · 2004
Cited alongside, same era.
Language models are few-shot learners, 2020, 2005.14165
Original
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2005

Then

Fixing weight decay regularization in adam
Original
Ilya Loshchilov and Frank Hutter · 2017
Later among the works it cites.
The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning
Original
Siyuan Ma, Raef Bassily, and Mikhail Belkin · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin · 2017
Later among the works it cites.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Later among the works it cites.
Generating wikipedia by summarizing long sequences
Original
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer · 2018

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2020

Scaling Laws for Autoregressive Generative Modeling

Henighan, Tom, Kaplan, Jared, Katz, Mor et al.

Understand

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving.

In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law.
The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains.
The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions.

Built on

Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720
Original
Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington · 1902
Earlier work this paper cites.
Generating long sequences with sparse transformers
Original
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever · 1904
Earlier work this paper cites.
Analysing mathematical reasoning abilities of neural models
Original
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli · 1904
Earlier work this paper cites.
One epoch is all you need, 2019, arXiv:1906.06669
Original
Aran Komatsuzaki · 1906
Earlier work this paper cites.
Scaling autoregressive video models, 2019, 1906.02634
Original
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit · 1906
Earlier work this paper cites.
Compositionality decomposed: how do neural networks generalise?, 2019, 1908.08351
Original
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni · 1908
Earlier work this paper cites.
A constructive prediction of the generalization error across scales, 2019, 1909.12673
Original
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit · 1909
Earlier work this paper cites.
Enhancing the transformer with explicit relational encoding for math problem solving, 2019, 1910.06611
Original
Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao · 1910
Earlier work this paper cites.
Scaling laws for neural language models, 2020, 2001.08361
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2001
Earlier work this paper cites.
Train large, then compress: Rethinking model size for efficient training and inference of transformers, 2020, 2002.11794
Original
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E. Gonzalez · 2002
Earlier work this paper cites.

Similar

The large learning rate phase of deep learning: the catapult mechanism, 2020, 2003.02218
Original
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari · 2003
Cited alongside, same era.
Recipes for building an open-domain chatbot, 2020, 2004.13637
Original
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston · 2004
Cited alongside, same era.
A neural scaling law from the dimension of the data manifold, 2020, 2004.10802
Original
Utkarsh Sharma and Jared Kaplan · 2004
Cited alongside, same era.
Language models are few-shot learners, 2020, 2005.14165
Original
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2005

Then

Fixing weight decay regularization in adam
Original
Ilya Loshchilov and Frank Hutter · 2017
Later among the works it cites.
The power of interpolation: Understanding the effectiveness of SGD in modern over-parametrized learning
Original
Siyuan Ma, Raef Bassily, and Mikhail Belkin · 2017
Later among the works it cites.
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin · 2017
Later among the works it cites.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Later among the works it cites.
Generating wikipedia by summarizing long sequences
Original
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer · 2018

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Cited alongside, same era.

Jukebox: A generative model for music, 2020, 2005.00341

Original

Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever · 2005

Cited alongside, same era.

On the predictability of pruning across scales, 2020, 2006.10621

Original

Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit · 2006

Cited alongside, same era.

The new data and new challenges in multimedia research

Original

Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li · 2015

Cited alongside, same era.

Pixel recurrent neural networks

Original

Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu · 2016

Cited alongside, same era.

A downsampled variant of imagenet as an alternative to the CIFAR datasets

Original

Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter · 2017

Cited alongside, same era.

Deep learning scaling is predictable, empirically, 2017, 1712.00409

Original

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou · 2017

Cited alongside, same era.

Later among the works it cites.

An empirical model of large-batch training, 2018, arXiv:1812.06162

Original

Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team · 2018

Later among the works it cites.

Neural discrete representation learning, 2018, 1711.00937

Original

Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu · 2018

Later among the works it cites.

Multimodal transformer for unaligned multimodal language sequences

Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov · 2019

Later among the works it cites.

Generative pretraining from pixels

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever · 2020

Closest in time.