Built on

Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020
Original
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al · 1909
Earlier work this paper cites.
Query-key normalization for transformers, 2020
Original
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen · 2010
Earlier work this paper cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Original
Taku Kudo and John Richardson · 2012
Earlier work this paper cites.
Distilling the knowledge in a neural network, 2015
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Neural combinatorial optimization with reinforcement learning
Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio · 2017
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.

Similar

Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Cited alongside, same era.
Primer: Searching for efficient transformers for language modeling
David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le · 2021
Cited alongside, same era.
Going deeper with image transformers
Original
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou · 2021
Cited alongside, same era.
Opt: Open pre-trained transformer language models, 2022
Original
Susan Zhang, Stephen Roller, Naman Goyal, et al · 2022
Cited alongside, same era.
Quantizable transformers: Removing outliers by helping attention heads do nothing
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort · 2023
Cited alongside, same era.

Roformer: Enhanced transformer with rotary position embedding
Original
Jianlin Su, Yu Lu, Shengfeng Pan, et al · 2023
Later among the works it cites.
Stable and low-precision training for large-scale vision-language models
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, et al · 2023
Later among the works it cites.
Stabilizing transformer training by preventing attention entropy collapse
Original
Shuangfei Zhai*, Tatiana Likhomanenko*, Etai Littwin*, Dan Busbridge*, Jason Ramapuram*, Yizhe Zhang, Jiatao Gu, and Josh M. Susskind · 2023
Later among the works it cites.
Palm: Scaling language modeling with pathways
Original
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al · 2024
Closest in time.
Transformer normalisation layers and the independence of semantic subspaces, 2024
Original

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…

2024

Rybakov, Oleg, Chrzanowski, Mike, Dykas, Peter et al.

Training stability of large language models(LLMs) is an important research topic.

Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learning rates to force models to diverge.
One of the sources of training instability is the growth of logits in attention layers.
We extend the focus of the previous work and look not only at the magnitude of the logits but at all outputs of linear layers in the Transformer block.

Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020
Original
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al · 1909
Earlier work this paper cites.
Query-key normalization for transformers, 2020
Original
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen · 2010
Earlier work this paper cites.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing
Original
Taku Kudo and John Richardson · 2012
Earlier work this paper cites.
Distilling the knowledge in a neural network, 2015
Original
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Original
Diederik P. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
Neural combinatorial optimization with reinforcement learning
Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio · 2017
Earlier work this paper cites.
Sgdr: Stochastic gradient descent with warm restarts
Ilya Loshchilov and Frank Hutter · 2017
Earlier work this paper cites.

Language models are unsupervised multitask learners
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever · 2019
Cited alongside, same era.
Primer: Searching for efficient transformers for language modeling
David So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le · 2021
Cited alongside, same era.
Going deeper with image transformers
Original
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou · 2021
Cited alongside, same era.
Opt: Open pre-trained transformer language models, 2022
Original
Susan Zhang, Stephen Roller, Naman Goyal, et al · 2022
Cited alongside, same era.
Quantizable transformers: Removing outliers by helping attention heads do nothing
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort · 2023
Cited alongside, same era.

Roformer: Enhanced transformer with rotary position embedding
Original
Jianlin Su, Yu Lu, Shengfeng Pan, et al · 2023
Later among the works it cites.
Stable and low-precision training for large-scale vision-language models
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, et al · 2023
Later among the works it cites.
Stabilizing transformer training by preventing attention entropy collapse
Original
Shuangfei Zhai*, Tatiana Likhomanenko*, Etai Littwin*, Dan Busbridge*, Jason Ramapuram*, Yizhe Zhang, Jiatao Gu, and Josh M. Susskind · 2023
Later among the works it cites.
Palm: Scaling language modeling with pathways
Original
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al · 2024
Closest in time.
Transformer normalisation layers and the independence of semantic subspaces, 2024
Original