Built on

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., Language models are few-shot learners , Advances in neural information processing systems 33
1901
Earlier work this paper cites.
Andrew M Saxe, James L McClelland, and Surya Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , arXiv preprint arXiv:1312.6120 (2013)
Original
2013
Earlier work this paper cites.
Diederik P Kingma and Jimmy Ba, Adam: A method for stochastic optimization , arXiv preprint arXiv:1412.6980 (2014)
Original
2014
Earlier work this paper cites.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need , ArXiv abs/1706.03762
Original
2017
Earlier work this paper cites.
Sanjeev Arora, Nadav Cohen, and Elad Hazan, On the optimization of deep networks: Implicit acceleration by overparameterization , International Conference on Machine Learning, PMLR, 2018, pp. 244–253
2018
Earlier work this paper cites.
Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach, On lazy training in differentiable programming , Neural Information Processing Systems, 2018
2018
Earlier work this paper cites.
Simon S Du, Wei Hu, and Jason D Lee, Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced , Advances in Neural Information Processing Systems 31
2018
Earlier work this paper cites.
Arthur Jacot, Franck Gabriel, and Clément Hongler, Neural tangent kernel: Convergence and generalization in neural networks , Advances in neural information processing systems 31
2018
Earlier work this paper cites.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski, Measuring the intrinsic dimension of objective landscapes , arXiv preprint arXiv:1804.08838 (2018)
Original
2018
Earlier work this paper cites.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo, Implicit regularization in deep matrix factorization , Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding , ArXiv abs/1810.04805
Original
2019
Earlier work this paper cites.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien, Implicit regularization of discrete gradient dynamics in linear neural networks , Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari, Limitations of lazy training of two-layers neural network , Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart, Disentangling feature and lazy learning in deep neural networks: an empirical study , ArXiv abs/1906.08034
Original
2019
Earlier work this paper cites.
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely, The implicit bias of depth: How incremental learning drives generalization , arXiv preprint arXiv:1909.12051 (2019)
Original
2019
Earlier work this paper cites.

Similar

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, Roberta: A robustly optimized bert pretraining approach , ArXiv abs/1907.11692
Original
2019
Cited alongside, same era.
Blake E. Woodworth, Suriya Gunasekar, J. Lee, Edward Moroshko, Pedro H. P. Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro, Kernel and rich regimes in overparametrized models , ArXiv abs/2002.09277
Original
2019
Cited alongside, same era.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta, Intrinsic dimensionality explains the effectiveness of language model fine-tuning , arXiv preprint arXiv:2012.13255 (2020)
Original
2020
Cited alongside, same era.
Francis Bach, Effortless optimization through gradient flows , Machine Learning Research Blog. https://francisbach. com/gradient-flows (2020)
2020
Cited alongside, same era.

Then

Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song, Learning single-index models with shallow neural networks , arXiv preprint arXiv:2210.15651 (2022)
Original
2022
Later among the works it cites.
Raphaël Berthier, Incremental learning in diagonal linear networks , arXiv preprint arXiv:2208.14673 (2022)
Original
2022
Later among the works it cites.
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion, Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs , arXiv preprint arXiv:2206.00939 (2022)
Original
2022
Later among the works it cites.
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi, Neural networks can learn representations with gradient descent , Conference on Learning Theory, PMLR, 2022, pp. 5413–5452
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2023

Transformers learn through gradual rank increase

Boix-Adsera, Enric, Littwin, Etai, Abbe, Emmanuel et al.

Understand

We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank.

We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization.
Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.

Built on

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., Language models are few-shot learners , Advances in neural information processing systems 33
1901
Earlier work this paper cites.
Andrew M Saxe, James L McClelland, and Surya Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , arXiv preprint arXiv:1312.6120 (2013)
Original
2013
Earlier work this paper cites.
Diederik P Kingma and Jimmy Ba, Adam: A method for stochastic optimization , arXiv preprint arXiv:1412.6980 (2014)
Original
2014
Earlier work this paper cites.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, Attention is all you need , ArXiv abs/1706.03762
Original
2017
Earlier work this paper cites.
Sanjeev Arora, Nadav Cohen, and Elad Hazan, On the optimization of deep networks: Implicit acceleration by overparameterization , International Conference on Machine Learning, PMLR, 2018, pp. 244–253
2018
Earlier work this paper cites.
Lénaïc Chizat, Edouard Oyallon, and Francis R. Bach, On lazy training in differentiable programming , Neural Information Processing Systems, 2018
2018
Earlier work this paper cites.
Simon S Du, Wei Hu, and Jason D Lee, Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced , Advances in Neural Information Processing Systems 31
2018
Earlier work this paper cites.
Arthur Jacot, Franck Gabriel, and Clément Hongler, Neural tangent kernel: Convergence and generalization in neural networks , Advances in neural information processing systems 31
2018
Earlier work this paper cites.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski, Measuring the intrinsic dimension of objective landscapes , arXiv preprint arXiv:1804.08838 (2018)
Original
2018
Earlier work this paper cites.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo, Implicit regularization in deep matrix factorization , Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding , ArXiv abs/1810.04805
Original
2019
Earlier work this paper cites.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien, Implicit regularization of discrete gradient dynamics in linear neural networks , Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari, Limitations of lazy training of two-layers neural network , Advances in Neural Information Processing Systems 32
2019
Earlier work this paper cites.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart, Disentangling feature and lazy learning in deep neural networks: an empirical study , ArXiv abs/1906.08034
Original
2019
Earlier work this paper cites.
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely, The implicit bias of depth: How incremental learning drives generalization , arXiv preprint arXiv:1909.12051 (2019)
Original
2019
Earlier work this paper cites.

Similar

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov, Roberta: A robustly optimized bert pretraining approach , ArXiv abs/1907.11692
Original
2019
Cited alongside, same era.
Blake E. Woodworth, Suriya Gunasekar, J. Lee, Edward Moroshko, Pedro H. P. Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro, Kernel and rich regimes in overparametrized models , ArXiv abs/2002.09277
Original
2019
Cited alongside, same era.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta, Intrinsic dimensionality explains the effectiveness of language model fine-tuning , arXiv preprint arXiv:2012.13255 (2020)
Original
2020
Cited alongside, same era.
Francis Bach, Effortless optimization through gradient flows , Machine Learning Research Blog. https://francisbach. com/gradient-flows (2020)
2020
Cited alongside, same era.

Then

Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song, Learning single-index models with shallow neural networks , arXiv preprint arXiv:2210.15651 (2022)
Original
2022
Later among the works it cites.
Raphaël Berthier, Incremental learning in diagonal linear networks , arXiv preprint arXiv:2208.14673 (2022)
Original
2022
Later among the works it cites.
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion, Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs , arXiv preprint arXiv:2206.00939 (2022)
Original
2022
Later among the works it cites.
Alexandru Damian, Jason Lee, and Mahdi Soltanolkotabi, Neural networks can learn representations with gradient descent , Conference on Learning Theory, PMLR, 2022, pp. 5413–5452
2022
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale , ArXiv abs/2010.11929

Original

2020

Cited alongside, same era.

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu, Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning , ArXiv abs/2012.09839

Original

2020

Cited alongside, same era.

Edward Moroshko, Suriya Gunasekar, Blake E. Woodworth, J. Lee, Nathan Srebro, and Daniel Soudry, Implicit bias in deep linear classification: Initialization scale vs training accuracy , ArXiv abs/2007.06738

Original

2020

Cited alongside, same era.

Noam Razin and Nadav Cohen, Implicit regularization in deep learning may not be explainable by norms , Advances in neural information processing systems 33

2020

Cited alongside, same era.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, Lora: Low-rank adaptation of large language models , arXiv preprint arXiv:2106.09685 (2021)

Original

2021

Cited alongside, same era.

Arthur Jacot, Francois Gaston Ged, Berfin Simsek, Clément Hongler, and Franck Gabriel, Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity , 2021

2021

Cited alongside, same era.

Paolo Milanesi, Hachem Kadri, S. Ayache, and Thierry Artières, Implicit regularization in deep tensor factorization , 2021 International Joint Conference on Neural Networks (IJCNN) (2021), 1–8

2021

Cited alongside, same era.

Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro, Quantifying the benefit of using differentiable learning over tangent kernels , Proceedings of the 38th International Conference on Machine Learning (Marina Meila and Tong Zhang, eds.), Proceedings of Machine Learning Research, vol. 139, PMLR, 18–24 Jul 2021, pp. 7379–7389

2021

Cited alongside, same era.

Spencer Frei, Gal Vardi, Peter L Bartlett, Nathan Srebro, and Wei Hu, Implicit bias in leaky relu networks trained on high-dimensional data , arXiv preprint arXiv:2210.07082 (2022)

Original

2022

Later among the works it cites.

Daesung Kim and Hye Won Chung, Rank-1 matrix completion with gradient descent and small random initialization , ArXiv abs/2212.09396

Original

2022

Later among the works it cites.

Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu, Neural networks efficiently learn low-dimensional representations with sgd , arXiv preprint arXiv:2209.14863 (2022)

Original

2022

Later among the works it cites.

Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon S Du, and Jason D Lee, Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing , arXiv preprint arXiv:2301.11500 (2023)

Original

2023

Closest in time.

Dylan Patel and Afzal Ahmad, Google “we have no moat, and neither does openai” , May 2023

2023

Closest in time.

Scott Pesme and Nicolas Flammarion, Saddle-to-saddle dynamics in diagonal linear networks , arXiv preprint arXiv:2304.00488 (2023)

Original

2023

Closest in time.

James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht, On the stepwise nature of self-supervised learning , arXiv preprint arXiv:2303.15438 (2023)

Original

2023

Closest in time.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto, Alpaca: A strong, replicable instruction-following model , Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html (2023)

2023

Closest in time.

Nadav Timor, Gal Vardi, and Ohad Shamir, Implicit regularization towards rank minimization in relu networks , International Conference on Algorithmic Learning Theory, PMLR, 2023, pp. 1429–1459

2023

Closest in time.

Jiawei Zhao, Yifei Zhang, Beidi Chen, Florian Schäfer, and Anima Anandkumar, Inrank: Incremental low-rank learning , arXiv preprint arXiv:2306.11250 (2023)

Original

2023

Closest in time.