HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position

Learning to represent programs with graphs

Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2018 · 2018

Earlier work this paper cites.

https://huggingface.co/codeparrot

CodeParrot. 2022 · 2022

Earlier work this paper cites.

Longbench: A bilingual, multitask benchmark for long context understanding

Original

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023 · 2023

Earlier work this paper cites.

Ntk-aware scaled rope allows llama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

bloc97. 2023 · 2023

Earlier work this paper cites.

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023 · 2023

Earlier work this paper cites.

Common crawl

Common Crawl. 2023 · 2023

Earlier work this paper cites.

Longnet: Scaling transformers to 1, 000, 000, 000 tokens

Original

Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. 2023 · 2023

Earlier work this paper cites.

Needle in a haystack - pressure testing llms

gkamradt. 2023 · 2023

Earlier work this paper cites.

https://platform.openai.com/docs/models/gpt-3-5

GPT-3.5. 2023 · 2023

Earlier work this paper cites.

Longcoder: A long-range pre-trained language model for code completion

Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian J. McAuley. 2023 · 2023

Cited alongside, same era.

Lm-infinite: Simple on-the-fly length generalization for large language models

Original

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2023 · 2023

Cited alongside, same era.

Giraffe: Adventures in expanding context lengths in llms

Original

Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. 2023 · 2023

Cited alongside, same era.

Yarn: Efficient context window extension of large language models

Original

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023 · 2023

Cited alongside, same era.

Code llama: Open foundation models for code

Effective long-context scaling of foundation models

Original

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. 2023 · 2023

Later among the works it cites.

Implant global and local hierarchy information to sequence based code representation models

Kechi Zhang, Zhuo Li, Zhi Jin, and Ge Li. 2023 · 2023

Later among the works it cites.

tree-sitter/tree-sitter: v0.21.0-pre-release-1

Max Brunsfeld, Andrew Hlynskyi, Amaan Qureshi, Patrick Thomson, Josh Vera, and et al. 2024 · 2024

Closest in time.

Two stones hit one bird: Bilevel positional encoding for better length extrapolation

Original

Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, and Liwei Wang. 2024 · 2024

Closest in time.

LLM maybe longlm: Self-extend LLM context window without tuning