Fetching the paper…

Reading the bibliography…

2024

Physics in Next-token Prediction

An, Hongjun, Song, Yiliang, Li, Xuelong

Original
PDF
Post on X

Understand

We discovered the underlying physics in Next-token Prediction (NTP).

We identified the law of information conservation within NTP and proposed the First Law of Information Capacity (IC-1), demonstrating that the essence of intelligence emergence in auto-regressive models is fundamentally a process of information transfer.
We also introduced Landauer's Principle into NTP, formulating the Second Law of Information Capacity (IC-2), which establishes the relationship between auto-regressive model training and energy consumption.
Additionally, we presented several corollaries, which hold practical significance for production practices.

Physics in Next-token Prediction · Around

Built on

A Mathematical Theory of Communication
Shannon, C. E · 1948
Earlier work this paper cites.
Irreversibility and heat generation in the computing process
Landauer, R · 1961
Earlier work this paper cites.
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., and Williams, R. J · 1986
Earlier work this paper cites.

Similar

Information preservation and weather forecasting for black holes, 2014
Hawking, S. W · 2014
Cited alongside, same era.
Scaling Laws for Neural Language Models, 2020
Kaplan, J., McCandlish, S., Henighan, T., et al · 2020
Cited alongside, same era.
Video distillation
Li, X. and Zhao, B · 2021
Cited alongside, same era.

Then

Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws, 2024
Allen-Zhu, Z. and Li, Y · 2024
Closest in time.
Scaling Laws for Precision, 2024
Kumar, T., Ankner, Z., Spector, B. F., et al · 2024
Closest in time.
Compression for AGI, 2023
Rae, J · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…