When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin · 2017

Earlier work this paper cites.

Bert: Pre-training of deep bidirectional transformers for language understanding

Original

Jacob Devlin · 2018

Earlier work this paper cites.

Self-attention with relative position representations

Original

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani · 2018

Earlier work this paper cites.

A study of bfloat16 for deep learning training

Original

Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, et al · 2019

Earlier work this paper cites.

Hellaswag: Can a machine really finish your sentence?

Original

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi · 2019

Earlier work this paper cites.

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush · 2020

Earlier work this paper cites.

Aligning ai with shared human values

Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt · 2021

Earlier work this paper cites.

Rethinking positional encoding in language pre-training

Guolin Ke, Di He, and Tie-Yan Liu · 2021

Earlier work this paper cites.

Sequence parallelism: Long sequence training from system perspective

Original

Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You · 2021

Earlier work this paper cites.

Roformer: Enhanced transformer with rotary position embedding

Original

Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu · 2021

Earlier work this paper cites.

Lifelong and continual learning dialogue systems

Original

Sahisnu Mazumder and Bing Liu · 2022

Earlier work this paper cites.

Gqa: Training generalized multi-query transformer models from multi-head checkpoints

Original

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai · 2023

Earlier work this paper cites.

Longbench: A bilingual, multitask benchmark for long context understanding

Original

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li · 2023

Earlier work this paper cites.

Lm-infinite: Simple on-the-fly length generalization for large language models

Original

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang · 2023

Earlier work this paper cites.

Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023

Original

Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He · 2023

Earlier work this paper cites.

Mistral 7b

Original

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al · 2023

Earlier work this paper cites.

Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degration, 2023

LocalLLaMA · 2023

Earlier work this paper cites.

Yarn: Efficient context window extension of large language models

Original

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole · 2023

Cited alongside, same era.

Randomized positional encodings boost length generalization of transformers

Original

Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness · 2023

Cited alongside, same era.

SlimPajama: A 627B token cleaned and deduplicated version of RedPajama

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey · 2023

Cited alongside, same era.

Llama 2: Open foundation and fine-tuned chat models, 2023

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and et al · 2023

Cited alongside, same era.

Efficient streaming language models with attention sinks

Original

When attention sink emerges in language models: An empirical view

Original

Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin · 2024

Closest in time.

Ruler: What’s the real context size of your long-context language models?

Original

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg · 2024

Closest in time.

Can perplexity reflect large language model’s ability in long text understanding?

Original

Yutong Hu, Quzhe Huang, Mingxu Tao, Chen Zhang, and Yansong Feng · 2024

Closest in time.

SWE-bench: Can language models resolve real-world github issues?

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan · 2024

Closest in time.

Llm maybe longlm: Self-extend llm context window without tuning