Understand
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML).
- In this paper we study trends in the most readily quantified factor - compute.
- We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months.
- Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months.
Built on
( \bibnodate
Klein, D · 2018
Earlier work this paper cites.
( \bibnodate
Baidu Research · 2022
Earlier work this paper cites.
( \bibnodate
GPT-Neo · 2022
Earlier work this paper cites.
( \bibnodate
J, T., Sejnowski & Rosenberg, C.R · 2022
Earlier work this paper cites.
Similar
( \bibnodate
Lieber, O., Sharir, O., Lenz, B. & Shoham, Y · 2022
Cited alongside, same era.
( \bibnodate
nad Mikhail Pavlov, A.R., Goh, G. & Gray, S · 2022
Cited alongside, same era.
( \bibnodate
Naver Corporation · 2022
Cited alongside, same era.
Then
( \bibnodate
OpenAI, Akkaya, I., Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B.Zhang, L · 2022
Closest in time.
( \bibnodate
Rae, J., Irving, G. & Weidinger, L · 2022
Closest in time.
( \bibnodate
Selfridge, O.G · 2022
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…