Built on

Shannon, C.E. A Mathematical Theory of Communication
1948
Earlier work this paper cites.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000
2000
Earlier work this paper cites.
Cover, T.M. and Thomas, J.A. Elements of Information Theory
2006
Earlier work this paper cites.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science , 411(29):2696–2711, 2010
2010
Earlier work this paper cites.
Giraldo, L.G.S., Rao, M., and Principe, J.C. Measures of entropy from data using infinitely divisible kernels
2014
Earlier work this paper cites.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Multitask Learners
2019
Earlier work this paper cites.

Similar

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. arXiv preprint arXiv:2001.08361, 2020
Original
2020
Cited alongside, same era.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022
Original
2022
Cited alongside, same era.
Tadros, T., Krishnan, G.P., Ramyaa, R., Bazhenov, M Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks
2022
Cited alongside, same era.
Skean, O., Osorio, J.K.H., Brockmeier, A.J., and Giraldo, L.G.S. DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies
Original
2023
Cited alongside, same era.

Then

Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A.C., Korbak, T., and Evans, O. The Reversal Curse: LLMs Trained on "A is B" Fail to Learn "B is A"
Original
2024
Later among the works it cites.
modded-nanogpt: Speedrunning the NanoGPT baseline
Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Vlado Boza, Jiacheng You, Franz Cesista, Braden Koszarsky, and @Grad62304977 · 2024
Later among the works it cites.
DeepSeek AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Original
2025
Closest in time.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv preprint arXiv:2504.13837 , 2025
Original
2025
Closest in time.
Fangyuan Yu. Scaling LLM pre-training with vocabulary curriculum. arXiv preprint arXiv:2502.17910, 2025

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

Memorization-Compression Cycles Improve Generalization

Yu, Fangyuan

Understand

We prove theoretically that generalization improves not only through data scaling but also by compressing internal representations.

To operationalize this insight, we introduce the Information Bottleneck Language Modeling (IBLM) objective, which reframes language modeling as a constrained optimization problem: minimizing representation entropy subject to optimal prediction performance.
Empirically, we observe an emergent memorization-compression cycle during LLM pretraining, evidenced by oscillation positive/negative gradient alignment between cross-entropy and Matrix-Based Entropy (MBE), a measure of representation entropy.
This pattern closely mirrors the predictive-compressive trade-off prescribed by IBLM and also parallels the biological alternation between awake learning and sleep consolidation.

Built on

Shannon, C.E. A Mathematical Theory of Communication
1948
Earlier work this paper cites.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000
2000
Earlier work this paper cites.
Cover, T.M. and Thomas, J.A. Elements of Information Theory
2006
Earlier work this paper cites.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science , 411(29):2696–2711, 2010
2010
Earlier work this paper cites.
Giraldo, L.G.S., Rao, M., and Principe, J.C. Measures of entropy from data using infinitely divisible kernels
2014
Earlier work this paper cites.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language Models are Unsupervised Multitask Learners
2019
Earlier work this paper cites.

Similar

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. arXiv preprint arXiv:2001.08361, 2020
Original
2020
Cited alongside, same era.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022
Original
2022
Cited alongside, same era.
Tadros, T., Krishnan, G.P., Ramyaa, R., Bazhenov, M Sleep-like unsupervised replay reduces catastrophic forgetting in artificial neural networks
2022
Cited alongside, same era.
Skean, O., Osorio, J.K.H., Brockmeier, A.J., and Giraldo, L.G.S. DiME: Maximizing Mutual Information by a Difference of Matrix-Based Entropies
Original
2023
Cited alongside, same era.

Then

Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A.C., Korbak, T., and Evans, O. The Reversal Curse: LLMs Trained on "A is B" Fail to Learn "B is A"
Original
2024
Later among the works it cites.
modded-nanogpt: Speedrunning the NanoGPT baseline
Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Vlado Boza, Jiacheng You, Franz Cesista, Braden Koszarsky, and @Grad62304977 · 2024
Later among the works it cites.
DeepSeek AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Original
2025
Closest in time.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv preprint arXiv:2504.13837 , 2025
Original
2025
Closest in time.
Fangyuan Yu. Scaling LLM pre-training with vocabulary curriculum. arXiv preprint arXiv:2502.17910, 2025

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. arXiv preprint arXiv:2309.10668, 2024

Original

2024

Cited alongside, same era.

Yu, F., Arora, H.S., and Johnson, M. Iterative Graph Alignment

Original

2024

Cited alongside, same era.

Tishby, N. and Zaslavsky, N. Deep Learning and the Information Bottleneck Principle

Original

Cited in the paper.

Oscar Skean and Md Rifat Arefin and Dan Zhao and Niket Patel and Jalal Naghiyev and Yann LeCun and Ravid Shwartz-Ziv Layer by Layer: Uncovering Hidden Representations in Language Models

Cited in the paper.

Ravid Shwartz-Ziv and Naftali Tishby Opening the Black Box of Deep Neural Networks via Information

Original

Cited in the paper.

Zeyuan Allen-Zhu and Yuanzhi Li Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

Original

Cited in the paper.

Yuzhen Huang and Jinghan Zhang and Zifei Shan and Junxian He Compression Represents Intelligence Linearly

Cited in the paper.

Oscar C González, Yury Sokolov, Giri P Krishnan, Jean Erik Delanois, Maxim Bazhenov Can sleep protect memorise from catastrophic forgetting?

Cited in the paper.

Original

2025

Closest in time.