Built on

“Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism”, 2020
Original
Mohammad Shoeybi et al · 1909
Earlier work this paper cites.
“Fast Transformer Decoding: One Write-Head is All You Need”, 2019
Original
Noam Shazeer · 1911
Earlier work this paper cites.
“Singular value decomposition for genome-wide expression data processing and modeling”
Orly Alter, Patrick. Brown and David Botstein · 2000
Earlier work this paper cites.
“Scaling Laws for Neural Language Models”, 2020
Original
Jared Kaplan et al · 2001
Earlier work this paper cites.
“The effective rank: A measure of effective dimensionality”
Olivier Roy and Martin Vetterli · 2007
Earlier work this paper cites.
“Measuring Massive Multitask Language Understanding”, 2021
Original
Dan Hendrycks et al · 2009
Earlier work this paper cites.
“Adam: A Method for Stochastic Optimization”
Original
Diederik. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
“TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, 2017
Original
Mandar Joshi et al · 2017
Earlier work this paper cites.
“Preconditioned Stochastic Gradient Descent”
Xi-Lin Li · 2017
Earlier work this paper cites.
“Preconditioner on Matrix Lie Group for SGD”, 2018
Original
Xi-Lin Li · 2018
Earlier work this paper cites.
“Decoupled Weight Decay Regularization”
Ilya Loshchilov and Frank Hutter · 2019
Earlier work this paper cites.
“ZeRO: Memory optimizations Toward Training Trillion Parameter Models”
Samyam Rajbhandari et al · 2020
Earlier work this paper cites.
“Program Synthesis with Large Language Models”, 2021
Original
Jacob Austin et al · 2021
Earlier work this paper cites.
“Evaluating Large Language Models Trained on Code”, 2021
Original
Mark Chen et al · 2021
Earlier work this paper cites.
“Training Verifiers to Solve Math Word Problems”, 2021
Original
Karl Cobbe et al · 2021
Earlier work this paper cites.

Similar

“Measuring Mathematical Problem Solving With the MATH Dataset”, 2021
Original
Dan Hendrycks et al · 2021
Cited alongside, same era.
“Training Compute-Optimal Large Language Models”, 2022
Original
Jordan Hoffmann et al · 2022
Cited alongside, same era.
“Black Box Lie Group Preconditioners for SGD”, 2022
Original
Xilin Li · 2022
Cited alongside, same era.
“Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them”, 2022
Original
Mirac Suzgun et al · 2022
Cited alongside, same era.
“C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models”, 2023
Original

Then

“CMMLU: Measuring massive multitask language understanding in Chinese”, 2024
Original
Haonan Li et al · 2024
Later among the works it cites.
“Stochastic Hessian Fittings with Lie Groups”, 2024
Original
Xi-Lin Li · 2024
Later among the works it cites.
“Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”
Hong Liu et al · 2024
Later among the works it cites.
“2 OLMo 2 Furious”
Original
Team OLMo et al · 2024
Later among the works it cites.
“GPT-4 Technical Report”, 2024
Original
OpenAI et al · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

Muon is Scalable for LLM Training

Liu, Jingyuan, Su, Jianlin, Yao, Xingcheng et al.

Understand

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven.

We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale.
These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning.
Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training.

Built on

“Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism”, 2020
Original
Mohammad Shoeybi et al · 1909
Earlier work this paper cites.
“Fast Transformer Decoding: One Write-Head is All You Need”, 2019
Original
Noam Shazeer · 1911
Earlier work this paper cites.
“Singular value decomposition for genome-wide expression data processing and modeling”
Orly Alter, Patrick. Brown and David Botstein · 2000
Earlier work this paper cites.
“Scaling Laws for Neural Language Models”, 2020
Original
Jared Kaplan et al · 2001
Earlier work this paper cites.
“The effective rank: A measure of effective dimensionality”
Olivier Roy and Martin Vetterli · 2007
Earlier work this paper cites.
“Measuring Massive Multitask Language Understanding”, 2021
Original
Dan Hendrycks et al · 2009
Earlier work this paper cites.
“Adam: A Method for Stochastic Optimization”
Original
Diederik. Kingma and Jimmy Ba · 2015
Earlier work this paper cites.
“TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension”, 2017
Original
Mandar Joshi et al · 2017
Earlier work this paper cites.
“Preconditioned Stochastic Gradient Descent”
Xi-Lin Li · 2017
Earlier work this paper cites.
“Preconditioner on Matrix Lie Group for SGD”, 2018
Original
Xi-Lin Li · 2018
Earlier work this paper cites.
“Decoupled Weight Decay Regularization”
Ilya Loshchilov and Frank Hutter · 2019
Earlier work this paper cites.
“ZeRO: Memory optimizations Toward Training Trillion Parameter Models”
Samyam Rajbhandari et al · 2020
Earlier work this paper cites.
“Program Synthesis with Large Language Models”, 2021
Original
Jacob Austin et al · 2021
Earlier work this paper cites.
“Evaluating Large Language Models Trained on Code”, 2021
Original
Mark Chen et al · 2021
Earlier work this paper cites.
“Training Verifiers to Solve Math Word Problems”, 2021
Original
Karl Cobbe et al · 2021
Earlier work this paper cites.

Similar

“Measuring Mathematical Problem Solving With the MATH Dataset”, 2021
Original
Dan Hendrycks et al · 2021
Cited alongside, same era.
“Training Compute-Optimal Large Language Models”, 2022
Original
Jordan Hoffmann et al · 2022
Cited alongside, same era.
“Black Box Lie Group Preconditioners for SGD”, 2022
Original
Xilin Li · 2022
Cited alongside, same era.
“Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them”, 2022
Original
Mirac Suzgun et al · 2022
Cited alongside, same era.
“C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models”, 2023
Original

Then

“CMMLU: Measuring massive multitask language understanding in Chinese”, 2024
Original
Haonan Li et al · 2024
Later among the works it cites.
“Stochastic Hessian Fittings with Lie Groups”, 2024
Original
Xi-Lin Li · 2024
Later among the works it cites.
“Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”
Hong Liu et al · 2024
Later among the works it cites.
“2 OLMo 2 Furious”
Original
Team OLMo et al · 2024
Later among the works it cites.
“GPT-4 Technical Report”, 2024
Original
OpenAI et al · 2024
Later among the works it cites.