Built on

K. You, M. Long, J. Wang, and M. Jordan, “How does learning rate decay help modern neural networks? arxiv 2019,” arXiv preprint arXiv:1908.01878 , 1908
Original
1908
Earlier work this paper cites.
A. Neelakantan, L. Vilnis, Q. V. Le, I. Sutskever et al. , “Adding gradient noise improves learning for very deep networks,” arXiv preprint arXiv:1511.06807 , 2015
Original
2015
Earlier work this paper cites.
N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in IEEE ITW , 2015, pp. 1–5
2015
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016
Original
2016
Earlier work this paper cites.
A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, “Deep variational information bottleneck,” arXiv preprint arXiv:1612.00410 , 2016
Original
2016
Earlier work this paper cites.
P. Goyal, P. Dollár, R. Girshick et al. , “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677 , 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Adv. Neural Inf. Process. Syst. , vol. 30, 2017
2017
Earlier work this paper cites.
L. N. Smith, “Cyclical learning rates for training neural networks,” in IEEE WACV . IEEE, 2017, pp. 464–472
2017
Earlier work this paper cites.
S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team, “An empirical model of large-batch training,” arXiv preprint arXiv:1812.06162 , 2018
Original
2018
Earlier work this paper cites.
N. Shazeer and M. Stern, “Adafactor: Adaptive learning rates with sublinear memory cost,” in ICML , 2018, pp. 4596–4604
2018
Earlier work this paper cites.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” 2018
2018
Earlier work this paper cites.
C. B. Clement, M. Bierbaum, K. P. O’Keeffe, and A. A. Alemi, “On the use of arxiv as a dataset,” 2019
2019
Earlier work this paper cites.
H. Husain, H.-H. Wu et al. , “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019
Original
2019
Earlier work this paper cites.
Z. Li and S. Arora, “An exponential learning rate schedule for deep learning,” arXiv preprint arXiv:1910.07454 , 2019
Original
2019
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT , 2019, pp. 4171–4186
2019
Earlier work this paper cites.
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?” 2019
2019
Earlier work this paper cites.
K. Sakaguchi, R. L. Bras et al. , “Winogrande: An adversarial winograd schema challenge at scale,” 2019
2019
Earlier work this paper cites.
Y. Bisk, R. Zellers et al. , “Piqa: Reasoning about physical commonsense in natural language,” 2019
2019
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan et al. , “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020
Original
2020
Earlier work this paper cites.
L. Gao, S. Biderman, S. Black, L. Golding et al. , “The pile: An 800gb dataset of diverse text for language modeling,” arXiv preprint arXiv:2101.00027 , 2020
Original
2020
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah et al. , “Language models are few-shot learners,” Adv. Neural Inf. Process. Syst. , vol. 33, pp. 1877–1901, 2020
2020
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee et al. , “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res , vol. 21, no. 140, pp. 1–67, 2020
2020
Earlier work this paper cites.
M. Tancik, P. Srinivasan et al. , “Fourier features let networks learn high frequency functions in low dimensional domains,” Adv. Neural Inf. Process. Syst. , vol. 33, pp. 7537–7547, 2020
2020
Earlier work this paper cites.
X. Wang, S. Oh, and C.-H. Rhee, “Eliminating sharp minima from sgd with truncated heavy-tailed noise,” arXiv preprint arXiv:2102.04297 , 2021
Original
2021
Earlier work this paper cites.
allenai. (2021)
2021
Earlier work this paper cites.
K. Cobbe, V. Kosaraju, M. Bavarian et al. , “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168 , 2021
Original
2021
Earlier work this paper cites.
J. W. Rae, S. Borgeaud, T. Cai et al. , “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446 , 2021
Original
2021
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” 2021
2021
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” 2021
2021
Earlier work this paper cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Earlier work this paper cites.
J. Austin, A. Odena, M. Nye et al. , “Program synthesis with large language models,” 2021
2021
Earlier work this paper cites.
N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen et al. , “The flores-101 evaluation benchmark for low-resource and multilingual machine translation,” 2021
2021
Earlier work this paper cites.
P. Izmailov, P. Kirichenko, N. Gruver, and A. G. Wilson, “On feature learning in the presence of spurious correlations,” Adv. Neural Inf. Process. Syst. , vol. 35, pp. 38 516–38 532, 2022
2022
Earlier work this paper cites.
Q. Jia, Y. Liu, H. Tang, and K. Q. Zhu, “In-sample curriculum learning by sequence completion for natural language generation,” arXiv preprint arXiv:2211.11297 , 2022
Original
2022
Earlier work this paper cites.
J. Gilmer, B. Ghorbani et al. , “A loss curvature perspective on training instabilities of deep learning models,” in ICLR , 2022
2022
Earlier work this paper cites.
J. Hoffmann, S. Borgeaud, A. Mensch et al. , “Training compute-optimal large language models (2022),” arXiv preprint arXiv:2203.15556 , 2022
Original
2022
Earlier work this paper cites.
D. Kocetkov, R. Li, L. Ben Allal, J. Li et al. , “The stack: 3 tb of permissively licensed source code,” Preprint , 2022
2022
Earlier work this paper cites.
S. Zhang, S. Roller, N. Goyal, M. Artetxe et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022
Original
2022
Earlier work this paper cites.
B. Workshop, T. L. Scao, A. Fan et al. , “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100 , 2022
Original
2022
Earlier work this paper cites.
S. Kreps, R. M. McCain, and M. Brundage, “All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation,” J EXP POLIT SCI. , vol. 9, no. 1, pp. 104–117, 2022
2022
Earlier work this paper cites.
M. Suzgun, N. Scales, N. Schärli et al. , “Challenging big-bench tasks and whether chain-of-thought can solve them,” 2022
2022
Earlier work this paper cites.
F. Cassano, J. Gouwar et al. , “Multipl-e: A scalable and extensible approach to benchmarking neural code generation,” 2022
2022
Earlier work this paper cites.
F. Shi, M. Suzgun, M. Freitag et al. , “Language models are multilingual chain-of-thought reasoners,” 2022
2022
Earlier work this paper cites.
X. Meng, Y. Cao, and D. Zou, “Per-example gradient regularization improves learning signals from noisy data,” arXiv preprint arXiv:2303.17940 , 2023
Original
2023
Earlier work this paper cites.
H. Jin, W. Wei, X. Wang, W. Zhang, and Y. Wu, “Rethinking learning rate tuning in the era of large language models,” in 2023 CogMI . IEEE, 2023, pp. 112–121
2023
Earlier work this paper cites.
Y. Huang, J. Xu, J. Lai et al. , “Advancing transformer architecture in long-context large language models: A comprehensive survey,” arXiv preprint arXiv:2311.12351 , 2023
Original
2023
Earlier work this paper cites.
A. Defazio, A. Cutkosky, H. Mehta, and K. Mishchenko, “Optimal linear decay learning rate schedules and further refinements,” arXiv preprint arXiv:2310.07831 , 2023
Original
2023
Earlier work this paper cites.
S. Chen, S. Wong, L. Chen, and Y. Tian, “Extending context window of large language models via positional interpolation,” arXiv preprint arXiv:2306.15595 , 2023
Original
2023
Earlier work this paper cites.
bloc97, “Ntk-aware scaled rope allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation,” Jun 2023, reddit post, r/LocalLLaMA
2023
Earlier work this paper cites.
B. Peng, J. Quesnelle, H. Fan, and E. Shippole, “Yarn: Efficient context window extension of large language models,” arXiv preprint arXiv:2309.00071 , 2023
Original
2023
Earlier work this paper cites.

Similar

W. Xiong, J. Liu et al. , “Effective long-context scaling of foundation models,” arXiv preprint arXiv:2309.16039 , 2023
Original
2023
Cited alongside, same era.
K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba, “Openwebmath: An open dataset of high-quality mathematical web text,” 2023
2023
Cited alongside, same era.
W. Lian, G. Wang, B. Goodson et al. , “Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification,” 2023
2023
Cited alongside, same era.
N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng et al. , “Enhancing chat language models by scaling high-quality instructional conversations,” arXiv preprint arXiv:2305.14233 , 2023
Original
2023
Cited alongside, same era.
L. Soldaini and K. Lo, “peS2o (Pretraining Efficiently on S2ORC) Dataset,” Allen Institute for AI, Tech. Rep., 2023, oDC-By, https://github.com/allenai/pes2o

Then

L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra, “Smollm-corpus,” 2024
2024
Later among the works it cites.
M. Naïr, K. Yamani, L. S. Lhadj, and R. Baghdadi, “Curriculum learning for small code language models,” arXiv preprint arXiv:2407.10194 , 2024
Original
2024
Later among the works it cites.
S. Chaudhry and A. Sharma, “Data distribution-based curriculum learning,” IEEE Access , 2024
2024
Later among the works it cites.
X. Sun, Y. Chen et al. , “Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent,” arXiv preprint arXiv:2411.02265 , 2024
Original
2024
Later among the works it cites.
A. Dubey, A. Jauhri et al. , “The llama 3 herd of models,” ArXiv , vol. abs/2407.21783, 2024
Original
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

Mid-Training of Large Language Models: A Survey

Mo, Kaixiang, Shi, Yuxin, Weng, Weiwei et al.

Understand

Large language models (LLMs) are typically developed through large-scale pre-training followed by task-specific fine-tuning.

Recent advances highlight the importance of an intermediate mid-training stage, where models undergo multiple annealing-style phases that refine data quality, adapt optimization schedules, and extend context length.
This stage mitigates diminishing returns from noisy tokens, stabilizes convergence, and expands model capability in late training.
Its effectiveness can be explained through gradient noise scale, the information bottleneck, and curriculum learning, which together promote generalization and abstraction.

Built on

K. You, M. Long, J. Wang, and M. Jordan, “How does learning rate decay help modern neural networks? arxiv 2019,” arXiv preprint arXiv:1908.01878 , 1908
Original
1908
Earlier work this paper cites.
A. Neelakantan, L. Vilnis, Q. V. Le, I. Sutskever et al. , “Adding gradient noise improves learning for very deep networks,” arXiv preprint arXiv:1511.06807 , 2015
Original
2015
Earlier work this paper cites.
N. Tishby and N. Zaslavsky, “Deep learning and the information bottleneck principle,” in IEEE ITW , 2015, pp. 1–5
2015
Earlier work this paper cites.
I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016
Original
2016
Earlier work this paper cites.
A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, “Deep variational information bottleneck,” arXiv preprint arXiv:1612.00410 , 2016
Original
2016
Earlier work this paper cites.
P. Goyal, P. Dollár, R. Girshick et al. , “Accurate, large minibatch sgd: Training imagenet in 1 hour,” arXiv preprint arXiv:1706.02677 , 2017
Original
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Adv. Neural Inf. Process. Syst. , vol. 30, 2017
2017
Earlier work this paper cites.
L. N. Smith, “Cyclical learning rates for training neural networks,” in IEEE WACV . IEEE, 2017, pp. 464–472
2017
Earlier work this paper cites.
S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team, “An empirical model of large-batch training,” arXiv preprint arXiv:1812.06162 , 2018
Original
2018
Earlier work this paper cites.
N. Shazeer and M. Stern, “Adafactor: Adaptive learning rates with sublinear memory cost,” in ICML , 2018, pp. 4596–4604
2018
Earlier work this paper cites.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” 2018
2018
Earlier work this paper cites.
C. B. Clement, M. Bierbaum, K. P. O’Keeffe, and A. A. Alemi, “On the use of arxiv as a dataset,” 2019
2019
Earlier work this paper cites.
H. Husain, H.-H. Wu et al. , “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019
Original
2019
Earlier work this paper cites.
Z. Li and S. Arora, “An exponential learning rate schedule for deep learning,” arXiv preprint arXiv:1910.07454 , 2019
Original
2019
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT , 2019, pp. 4171–4186
2019
Earlier work this paper cites.
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?” 2019
2019
Earlier work this paper cites.
K. Sakaguchi, R. L. Bras et al. , “Winogrande: An adversarial winograd schema challenge at scale,” 2019
2019
Earlier work this paper cites.
Y. Bisk, R. Zellers et al. , “Piqa: Reasoning about physical commonsense in natural language,” 2019
2019
Earlier work this paper cites.
J. Kaplan, S. McCandlish, T. Henighan et al. , “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361 , 2020
Original
2020
Earlier work this paper cites.
L. Gao, S. Biderman, S. Black, L. Golding et al. , “The pile: An 800gb dataset of diverse text for language modeling,” arXiv preprint arXiv:2101.00027 , 2020
Original
2020
Earlier work this paper cites.
T. Brown, B. Mann, N. Ryder, M. Subbiah et al. , “Language models are few-shot learners,” Adv. Neural Inf. Process. Syst. , vol. 33, pp. 1877–1901, 2020
2020
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee et al. , “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res , vol. 21, no. 140, pp. 1–67, 2020
2020
Earlier work this paper cites.
M. Tancik, P. Srinivasan et al. , “Fourier features let networks learn high frequency functions in low dimensional domains,” Adv. Neural Inf. Process. Syst. , vol. 33, pp. 7537–7547, 2020
2020
Earlier work this paper cites.
X. Wang, S. Oh, and C.-H. Rhee, “Eliminating sharp minima from sgd with truncated heavy-tailed noise,” arXiv preprint arXiv:2102.04297 , 2021
Original
2021
Earlier work this paper cites.
allenai. (2021)
2021
Earlier work this paper cites.
K. Cobbe, V. Kosaraju, M. Bavarian et al. , “Training verifiers to solve math word problems,” arXiv preprint arXiv:2110.14168 , 2021
Original
2021
Earlier work this paper cites.
J. W. Rae, S. Borgeaud, T. Cai et al. , “Scaling language models: Methods, analysis & insights from training gopher,” arXiv preprint arXiv:2112.11446 , 2021
Original
2021
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” 2021
2021
Earlier work this paper cites.
D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, “Measuring mathematical problem solving with the math dataset,” 2021
2021
Earlier work this paper cites.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto et al. , “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
Original
2021
Earlier work this paper cites.
J. Austin, A. Odena, M. Nye et al. , “Program synthesis with large language models,” 2021
2021
Earlier work this paper cites.
N. Goyal, C. Gao, V. Chaudhary, P.-J. Chen et al. , “The flores-101 evaluation benchmark for low-resource and multilingual machine translation,” 2021
2021
Earlier work this paper cites.
P. Izmailov, P. Kirichenko, N. Gruver, and A. G. Wilson, “On feature learning in the presence of spurious correlations,” Adv. Neural Inf. Process. Syst. , vol. 35, pp. 38 516–38 532, 2022
2022
Earlier work this paper cites.
Q. Jia, Y. Liu, H. Tang, and K. Q. Zhu, “In-sample curriculum learning by sequence completion for natural language generation,” arXiv preprint arXiv:2211.11297 , 2022
Original
2022
Earlier work this paper cites.
J. Gilmer, B. Ghorbani et al. , “A loss curvature perspective on training instabilities of deep learning models,” in ICLR , 2022
2022
Earlier work this paper cites.
J. Hoffmann, S. Borgeaud, A. Mensch et al. , “Training compute-optimal large language models (2022),” arXiv preprint arXiv:2203.15556 , 2022
Original
2022
Earlier work this paper cites.
D. Kocetkov, R. Li, L. Ben Allal, J. Li et al. , “The stack: 3 tb of permissively licensed source code,” Preprint , 2022
2022
Earlier work this paper cites.
S. Zhang, S. Roller, N. Goyal, M. Artetxe et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022
Original
2022
Earlier work this paper cites.
B. Workshop, T. L. Scao, A. Fan et al. , “Bloom: A 176b-parameter open-access multilingual language model,” arXiv preprint arXiv:2211.05100 , 2022
Original
2022
Earlier work this paper cites.
S. Kreps, R. M. McCain, and M. Brundage, “All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation,” J EXP POLIT SCI. , vol. 9, no. 1, pp. 104–117, 2022
2022
Earlier work this paper cites.
M. Suzgun, N. Scales, N. Schärli et al. , “Challenging big-bench tasks and whether chain-of-thought can solve them,” 2022
2022
Earlier work this paper cites.
F. Cassano, J. Gouwar et al. , “Multipl-e: A scalable and extensible approach to benchmarking neural code generation,” 2022
2022
Earlier work this paper cites.
F. Shi, M. Suzgun, M. Freitag et al. , “Language models are multilingual chain-of-thought reasoners,” 2022
2022
Earlier work this paper cites.
X. Meng, Y. Cao, and D. Zou, “Per-example gradient regularization improves learning signals from noisy data,” arXiv preprint arXiv:2303.17940 , 2023
Original
2023
Earlier work this paper cites.
H. Jin, W. Wei, X. Wang, W. Zhang, and Y. Wu, “Rethinking learning rate tuning in the era of large language models,” in 2023 CogMI . IEEE, 2023, pp. 112–121
2023
Earlier work this paper cites.
Y. Huang, J. Xu, J. Lai et al. , “Advancing transformer architecture in long-context large language models: A comprehensive survey,” arXiv preprint arXiv:2311.12351 , 2023
Original
2023
Earlier work this paper cites.
A. Defazio, A. Cutkosky, H. Mehta, and K. Mishchenko, “Optimal linear decay learning rate schedules and further refinements,” arXiv preprint arXiv:2310.07831 , 2023
Original
2023
Earlier work this paper cites.
S. Chen, S. Wong, L. Chen, and Y. Tian, “Extending context window of large language models via positional interpolation,” arXiv preprint arXiv:2306.15595 , 2023
Original
2023
Earlier work this paper cites.
bloc97, “Ntk-aware scaled rope allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation,” Jun 2023, reddit post, r/LocalLLaMA
2023
Earlier work this paper cites.
B. Peng, J. Quesnelle, H. Fan, and E. Shippole, “Yarn: Efficient context window extension of large language models,” arXiv preprint arXiv:2309.00071 , 2023
Original
2023
Earlier work this paper cites.

Similar

W. Xiong, J. Liu et al. , “Effective long-context scaling of foundation models,” arXiv preprint arXiv:2309.16039 , 2023
Original
2023
Cited alongside, same era.
K. Paster, M. D. Santos, Z. Azerbayev, and J. Ba, “Openwebmath: An open dataset of high-quality mathematical web text,” 2023
2023
Cited alongside, same era.
W. Lian, G. Wang, B. Goodson et al. , “Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification,” 2023
2023
Cited alongside, same era.
N. Ding, Y. Chen, B. Xu, Y. Qin, Z. Zheng et al. , “Enhancing chat language models by scaling high-quality instructional conversations,” arXiv preprint arXiv:2305.14233 , 2023
Original
2023
Cited alongside, same era.
L. Soldaini and K. Lo, “peS2o (Pretraining Efficiently on S2ORC) Dataset,” Allen Institute for AI, Tech. Rep., 2023, oDC-By, https://github.com/allenai/pes2o

Then

L. Ben Allal, A. Lozhkov, G. Penedo, T. Wolf, and L. von Werra, “Smollm-corpus,” 2024
2024
Later among the works it cites.
M. Naïr, K. Yamani, L. S. Lhadj, and R. Baghdadi, “Curriculum learning for small code language models,” arXiv preprint arXiv:2407.10194 , 2024
Original
2024
Later among the works it cites.
S. Chaudhry and A. Sharma, “Data distribution-based curriculum learning,” IEEE Access , 2024
2024
Later among the works it cites.
X. Sun, Y. Chen et al. , “Hunyuan-large: An open-source moe model with 52 billion activated parameters by tencent,” arXiv preprint arXiv:2411.02265 , 2024
Original
2024
Later among the works it cites.
A. Dubey, A. Jauhri et al. , “The llama 3 herd of models,” ArXiv , vol. abs/2407.21783, 2024
Original
2024

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

2023

Cited alongside, same era.

G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru et al. , “The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only,” arXiv preprint arXiv:2306.01116 , 2023

Original

2023

Cited alongside, same era.

Z. Luo, C. Xu, P. Zhao et al. , “Wizardcoder: Empowering code large language models with evol-instruct,” 2023

2023

Cited alongside, same era.

W. Lian, B. Goodson, E. Pentland, A. Cook, C. Vong, and ”Teknium”, “Openorca: An open dataset of gpt augmented flan reasoning traces,” https://https://huggingface.co/datasets/Open-Orca/OpenOrca , 2023

2023

Cited alongside, same era.

L. Yu, W. Jiang, H. Shi, J. Yu et al. , “Metamath: Bootstrap your own mathematical questions for large language models,” arXiv preprint arXiv:2309.12284 , 2023

Original

2023

Cited alongside, same era.

C. Li, Z. Yuan, H. Yuan, G. Dong et al. , “Mugglemath: Assessing the impact of query and response augmentation on math reasoning,” arXiv preprint arXiv:2310.05506 , 2023

Original

2023

Cited alongside, same era.

T. Nguyen, C. Van Nguyen, V. D. Lai, H. Man, N. T. Ngo, F. Dernoncourt, R. A. Rossi, and T. H. Nguyen, “Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages,” arXiv preprint arXiv:2309.09400 , 2023

Original

2023

Cited alongside, same era.

Z. Shen, T. Tao et al. , “Slimpajama-dc: Understanding data combinations for llm training,” arXiv preprint arXiv:2309.10818 , 2023

Original

2023

Cited alongside, same era.