Built on

The information bottleneck method
Naftali Tishby, Fernando C Pereira, and William Bialek · 2000
Earlier work this paper cites.
Deep learning and the information bottleneck principle
Naftali Tishby and Noga Zaslavsky · 2015
Earlier work this paper cites.
On linear stability of sgd and input-smoothness of neural networks
Chao Ma and Lexing Ying · 2021
Earlier work this paper cites.
Information theory with kernel methods
Francis Bach · 2022
Earlier work this paper cites.
Hidden progress in deep learning: Sgd learns parities near the computational limit
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang · 2022
Earlier work this paper cites.
Grokking: Generalization beyond overfitting on small algorithmic datasets
Original
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra · 2022
Earlier work this paper cites.

Similar

The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon
Original
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind · 2022
Cited alongside, same era.
Grokking phase transitions in learning local rules with gradient descent
Original
Bojan Žunkovič and Enej Ilievski · 2022
Cited alongside, same era.
Unifying grokking and double descent
Original
Xander Davies, Lauro Langosco, and David Krueger · 2023
Cited alongside, same era.
Grokking modular arithmetic
Original
Andrey Gromov · 2023
Cited alongside, same era.
A tale of two circuits: Grokking as competition of sparse and dense subnetworks

Then

Progress measures for grokking via mechanistic interpretability
Original
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt · 2023
Closest in time.
Predicting grokking long before it happens: A look into the loss landscape of models which grok
Original
Pascal Notsawo Jr, Hattie Zhou, Mohammad Pezeshki, Irina Rish, Guillaume Dumas, et al · 2023
Closest in time.
Dime: Maximizing mutual information by a difference of matrix-based entropies
Original
Oscar Skean, Jhoan Keider Hoyos Osorio, Austin J Brockmeier, and Luis Gonzalo Sanchez Giraldo · 2023
Closest in time.
Information flow in self-supervised learning
Original
Zhiquan Tan, Jingqin Yang, Weiran Huang, Yang Yuan, and Yifan Zhang · 2023
Closest in time.
Explaining grokking through circuit efficiency

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Understanding Grokking Through A Robustness Viewpoint

Tan, Zhiquan, Huang, Weiran

Understand

Recently, an interesting phenomenon called grokking has gained much attention, where generalization occurs long after the models have initially overfitted the training data.

We try to understand this seemingly strange phenomenon through the robustness of the neural network.
From a robustness perspective, we show that the popular $l_2$ weight norm (metric) of the neural network is actually a sufficient condition for grokking.
Based on the previous observations, we propose perturbation-based methods to speed up the generalization process.

Built on

The information bottleneck method
Naftali Tishby, Fernando C Pereira, and William Bialek · 2000
Earlier work this paper cites.
Deep learning and the information bottleneck principle
Naftali Tishby and Noga Zaslavsky · 2015
Earlier work this paper cites.
On linear stability of sgd and input-smoothness of neural networks
Chao Ma and Lexing Ying · 2021
Earlier work this paper cites.
Information theory with kernel methods
Francis Bach · 2022
Earlier work this paper cites.
Hidden progress in deep learning: Sgd learns parities near the computational limit
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang · 2022
Earlier work this paper cites.
Grokking: Generalization beyond overfitting on small algorithmic datasets
Original
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra · 2022
Earlier work this paper cites.