The Epochal Sawtooth Phenomenon: Unveiling Training Loss Oscillations in Adam and Other Optimizers

Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. Ussr computational mathematics and mathematical physics 4

1964

Earlier work this paper cites.

Nesterov, Y.: A method for solving the convex programming problem with convergence rate o ⁡ ( 1 / k 2 ) o(1/k^{2}) . In: Dokl Akad Nauk Sssr, vol. 269, p. 543 (1983)

1983

Earlier work this paper cites.

Bottou, L., Bousquet, O.: The tradeoffs of large scale learning. Advances in neural information processing systems 20

2007

Earlier work this paper cites.

Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177–186 (2010). Springer

2010

Earlier work this paper cites.

Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research 12

2011

Earlier work this paper cites.

Tieleman, T., Hinton, G.: Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning. COURSERA Neural Networks Mach. Learn 17

2012

Earlier work this paper cites.

Kingma, D.P.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) 2015 (2014). ICLR

2014

Earlier work this paper cites.

Ruder, S.: An overview of gradient descent optimization algorithms. Vestnik komp iuternykh i informatsionnykh tekhnologii (2016)

2016

Cited alongside, same era.

Goodfellow, I.J., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge, MA, USA (2016). http://www.deeplearningbook.org

2016

Cited alongside, same era.

Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472 (2017). IEEE

2017

Cited alongside, same era.

Wilson, A.C., Roelofs, R., Stern, M., Srebro, N., Recht, B.: The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems 30

2017

Cited alongside, same era.

Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., Wilson, A.G.: Averaging weights leads to wider optima and better generalization. In: 34th Conference on Uncertainty in Artificial Intelligence 2018, pp. 876–885 (2018). Association For Uncertainty in Artificial Intelligence (AUAI)

Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2019)

Original

2019

Later among the works it cites.

Goyal, R.: WikiText Dataset. Accessed: January 16, 2025 (2021). https://www.kaggle.com/datasets/rohitgr/wikitext

2021

Later among the works it cites.

Liu, Q., Ma, W.: Training Loss Pattern Analysis. Accessed: January 16, 2025 (2023). https://github.com/qiliuchn/training-loss-pattern

2023

Later among the works it cites.

Hugging Face Community: Trainer’s Step Loss Drops Sharply After Each Epoch. Accessed: January 16, 2025 (2025). https://discuss.huggingface.co/t/trainers-step-loss-always-drops-sharply-after-each-epoch-regardless-of-model-data/27131

2025

Closest in time.

Hugging Face Community: Issue 18730: Unexpected Behavior in Transformers Trainer. Accessed: January 16, 2025 (2025). https://github.com/huggingface/transformers/issues/18730