From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models

Megatron-lm: Training multi-billion parameter language models using model parallelism

Original

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019 · 1909

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020 · 2009

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Measuring mathematical problem solving with the math dataset

Original

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 · 2021

Earlier work this paper cites.

Lora: Low-rank adaptation of large language models

Original

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021 · 2021

Earlier work this paper cites.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Earlier work this paper cites.

Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

bloc97. 2023 · 2023

Earlier work this paper cites.

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023 · 2023

Earlier work this paper cites.

Dynamically scaled rope further increases performance of long context llama with zero fine-tuning

emozilla. 2023 · 2023

Earlier work this paper cites.

Glaivecodeassistant

Glaive-AI. 2023 · 2023

Earlier work this paper cites.

Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification

Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023 · 2023

Earlier work this paper cites.

Wizardcoder: Empowering code large language models with evol-instruct

Original

Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023 · 2023

Earlier work this paper cites.

Landmark attention: Random-access infinite context length for transformers

Original

Amirkeivan Mohtashami and Martin Jaggi. 2023 · 2023

Earlier work this paper cites.

Orca: Progressive learning from complex explanation traces of gpt-4

Original

Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023 · 2023

Earlier work this paper cites.

Gpteacher-general-instruct

Teknium. 2023 · 2023

Earlier work this paper cites.

ShareGPT-Vicuna

The-Vicuna-Team. 2023 · 2023

Cited alongside, same era.

Retrieval meets long context large language models

Original

Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023 · 2023

Cited alongside, same era.

Metamath: Bootstrap your own mathematical questions for large language models

Original

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023 · 2023

Cited alongside, same era.

Mammoth: Building math generalist models through hybrid instruction tuning

Original

Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2023 · 2023

Cited alongside, same era.

Training-free long-context scaling of large language models

Original

Openai o1 system card

Original

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024 · 2024

Later among the works it cites.

A controlled study on long context extension and generalization in llms

Original

Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M Rush. 2024 · 2024

Later among the works it cites.

Orca-math: Unlocking the potential of slms in grade school math

Original

Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024 · 2024

Later among the works it cites.

Gpt-4o mini: advancing cost-efficient intelligence

OpenAI. 2024 · 2024

Later among the works it cites.

Llama 3 gradient: A series of long context models

Leonid Pekelis, Michael Feil, Forrest Moret, Mark Huang, and Tiffany Peng. 2024 · 2024