Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition

Attention is all you need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I · 2017

Earlier work this paper cites.

Reconciling modern machine-learning practice and the classical bias–variance trade-off

Belkin, M., Hsu, D., Ma, S., and Mandal, S · 2019

Earlier work this paper cites.

Decoupled weight decay regularization

Loshchilov, I. and Hutter, F · 2019

Earlier work this paper cites.

Language models are few-shot learners

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D · 2020

Earlier work this paper cites.

Deep double descent: Where bigger models and more data hurt

Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I · 2020

Earlier work this paper cites.

Transformer feed-forward layers are key-value memories

Geva, M., Schuster, R., Berant, J., and Levy, O · 2021

Earlier work this paper cites.

Understanding deep learning (still) requires rethinking generalization

Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O · 2021

Earlier work this paper cites.

Unifying grokking and double descent

Davies, X., Langosco, L., and Krueger, D · 2022

Earlier work this paper cites.

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

Fedus, W., Zoph, B., and Shazeer, N · 2022

Cited alongside, same era.

Towards understanding grokking: An effective theory of representation learning

Liu, Z., Kitouni, O., Nolte, N., Michaud, E. J., Tegmark, M., and Williams, M · 2022

Cited alongside, same era.

Grokking: Generalization beyond overfitting on small algorithmic datasets

Original

Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V · 2022

Cited alongside, same era.

The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon

Original

Thilak, V., Littwin, E., Zhai, S., Saremi, O., Paiss, R., and Susskind, J. M · 2022

Cited alongside, same era.

Chain-of-thought prompting elicits reasoning in large language models

Wei, J., Wang, X., Schuurmans, D., Bosma, M., ichter, b., Xia, F., Chi, E., Le, Q. V., and Zhou, D · 2022

Predicting emergent abilities with infinite resolution evaluation, 2023

Hu, S., Liu, X., Han, X., Zhang, X., He, C., Zhao, W., Lin, Y., Ding, N., Ou, Z., Zeng, G., Liu, Z., and Sun, M · 2023

Later among the works it cites.

Omnigrok: Grokking beyond algorithmic data

Liu, Z., Michaud, E. J., and Tegmark, M · 2023

Later among the works it cites.

The quantization model of neural scaling

Michaud, E. J., Liu, Z., Girit, U., and Tegmark, M · 2023

Later among the works it cites.

Progress measures for grokking via mechanistic interpretability

Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J · 2023

Later among the works it cites.

Predicting grokking long before it happens: A look into the loss landscape of models which grok

Original

Notsawo Jr., P. T., Zhou, H., Pezeshki, M., Rish, I., and Dumas, G · 2023

Later among the works it cites.

Are emergent abilities of large language models a mirage?