Fetching the paper…

Reading the bibliography…

2023

Grokking modular arithmetic

Gromov, Andrey

Original
PDF
Post on X

Understand

We present a simple neural network that can learn modular arithmetic tasks and exhibits a sudden jump in generalization known as ``grokking''.

Concretely, we present (i) fully-connected two-layer networks that exhibit grokking on various modular arithmetic tasks under vanilla gradient descent with the MSE loss function in the absence of any regularization; (ii) evidence that grokking modular arithmetic corresponds to learning specific feature maps whose structure is determined by the task; (iii) analytic expressions for the weights -- and thus for the feature maps -- that solve a large class of modular arithmetic tasks; and (iv) evidence that these feature maps are also found by vanilla gradient descent as well as AdamW, thereby establishing complete interpretability of the representations learnt by the network.

Grokking modular arithmetic · Around

Built on

Distinct types of eigenvector localization in networks
Romualdo Pastor-Satorras and Claudio Castellano · 2016
Earlier work this paper cites.
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clément Hongler · 2018
Earlier work this paper cites.
A mean field view of the landscape of two-layers neural networks
Mei Song, Andrea Montanari, and P Nguyen · 2018
Earlier work this paper cites.
Modern condensed matter physics
Steven M Girvin and Kun Yang · 2019
Earlier work this paper cites.
Wide neural networks of any depth evolve as linear models under gradient descent
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington · 2019
Earlier work this paper cites.

Similar

The large learning rate phase of deep learning: the catapult mechanism
Original
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari · 2020
Cited alongside, same era.
Feature learning in infinite-width neural networks
Original
Greg Yang and Edward J Hu · 2020
Cited alongside, same era.
How to quantify fields or textures? a guide to the scattering transform
Original
Sihao Cheng and Brice Ménard · 2021
Cited alongside, same era.
The principles of deep learning theory
Original
Daniel A Roberts, Sho Yaida, and Boris Hanin · 2021
Cited alongside, same era.
Hidden progress in deep learning: Sgd learns parities near the computational limit
Original
Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang · 2022
Cited alongside, same era.
Towards understanding grokking: An effective theory of representation learning
Original
Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J Michaud, Max Tegmark, and Mike Williams
Cited in the paper.
Omnigrok: Grokking beyond algorithmic data
Original
Ziming Liu, Eric J Michaud, and Max Tegmark
Cited in the paper.

Then

A mechanistic interpretability analysis of grokking
Neel Nanda and Tom Lieberum · 2022
Later among the works it cites.
Grokking: Generalization beyond overfitting on small algorithmic datasets
Original
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra · 2022
Later among the works it cites.
The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon
Original
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind · 2022
Later among the works it cites.
Grokking phase transitions in learning local rules with gradient descent
Original
Bojan Žunkovič and Enej Ilievski · 2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…