Fetching the paper…

Reading the bibliography…

2023

Weight Norm Control

Loshchilov, Ilya

Original
PDF
Post on X

Understand

We note that decoupled weight decay regularization is a particular case of weight norm control where the target norm of weights is set to 0.

Any optimization method (e.g., Adam) which uses decoupled weight decay regularization (respectively, AdamW) can be viewed as a particular case of a more general algorithm with weight norm control (respectively, AdamWN).
We argue that setting the target norm of weights to 0 can be suboptimal and other target norm values can be considered.
For instance, any training run where AdamW achieves a particular norm of weights can be challenged by AdamWN scheduled to achieve a comparable norm of weights.

Weight Norm Control · Around

Built on

Comparing biases for minimal network construction with back-propagation
Stephen José Hanson and Lorien Y Pratt · 1988
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik Kingma and Jimmy Ba · 2014
Earlier work this paper cites.
Dropout: A simple way to prevent neural networks from overfitting
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov · 2014
Earlier work this paper cites.

Similar

Layer normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton · 2016
Cited alongside, same era.
Weight normalization: A simple reparameterization to accelerate training of deep neural networks
Tim Salimans and Durk P Kingma · 2016
Cited alongside, same era.
Language models are unsupervised multitask learners
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2018
Cited alongside, same era.

Then

Openwebtext corpus
Aaron Gokaslan and Vanya Cohen · 2019
Later among the works it cites.
Decoupled weight decay regularization
Ilya Loshchilov and Frank Hutter · 2019
Later among the works it cites.
New horizons in parameter regularization: A constraint approach
Jörg K. H. Franke, Michael Hefenbrock, Gregor Koehler, and Frank Hutter · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…