Fetching the paper…

Reading the bibliography…

2024

Chinchilla Scaling: A replication attempt

Besiroglu, Tamay, Erdil, Ege, Barnett, Matthew et al.

Original
PDF
Post on X

Understand

Hoffmann et al.

(2022) propose three methods for estimating a compute-optimal scaling law.
We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots.
We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500.

Chinchilla Scaling: A replication attempt · Around

Built on

The large-sample distribution of the likelihood ratio for testing composite hypotheses
Wilks, S. S. (1938) · 1938
Earlier work this paper cites.
Training compute-optimal large language models
Original
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · 2022
Earlier work this paper cites.
Palm 2 technical report
Original
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · 2023
Earlier work this paper cites.

Similar

Gemini: a family of highly capable multimodal models
Original
Google, G. T. (2023) · 2023
Cited alongside, same era.
Beyond chinchilla-optimal: Accounting for inference in language model scaling laws
Original
Sardana, N. and Frankle, J. (2023) · 2023
Cited alongside, same era.
Deepseek llm: Scaling open-source language models with longtermism
Original
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024) · 2024
Cited alongside, same era.

Then

A dynamical model of neural scaling laws
Original
Bordelon, B., Atanasov, A., and Pehlevan, C. (2024) · 2024
Closest in time.
The quantization model of neural scaling
Michaud, E., Liu, Z., Girit, U., and Tegmark, M. (2024) · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…