Fetching the paper…

Reading the bibliography…

2024

Time Matters: Scaling Laws for Any Budget

Inbar, Itay, Sernau, Luke

Original
PDF
Post on X

Understand

A primary cost driver for training large models is wall-clock training time.

We show that popular time estimates based on FLOPs are poor estimates, and construct a more accurate proxy based on memory copies.
This allows us to accurately estimate the training speed of a transformer model from its hyperparameters.
Combined with a scaling law curve like Chinchilla, this allows us to accurately predict the final loss of a model from a simple equation.

Time Matters: Scaling Laws for Any Budget · Around

Built on

Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017
Earlier work this paper cites.
Scaling laws for neural language models
Original
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Earlier work this paper cites.

Similar

Documenting large webtext corpora: A case study on the colossal clean crawled corpus
Original
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner · 2021
Cited alongside, same era.
Training compute-optimal large language models
Original
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al · 2022
Cited alongside, same era.

Then

Chinchilla scaling: A replication attempt
Original
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…