Fetching the paper…

Reading the bibliography…

2024

Scaling Granite Code Models to 128K Context

Stallone, Matt, Saxena, Vaibhav, Karlinsky, Leonid et al.

Original
PDF
Post on X

Understand

This paper introduces long-context Granite code models that support effective context windows of up to 128K tokens.

Our solution for scaling context length of Granite 3B/8B code models from 2K/4K to 128K consists of a light-weight continual pretraining by gradually increasing its RoPE base frequency with repository-level file packing and length-upsampled long-context data.
Additionally, we also release instruction-tuned models with long-context support which are derived by further finetuning the long context base models on a mix of permissively licensed short and long-context instruction-response pairs.
While comparing to the original short-context Granite code models, our long-context models achieve significant improvements on long-context tasks without any noticeable performance degradation on regular code completion benchmarks (e.g., HumanEval).

Scaling Granite Code Models to 128K Context · Around

Built on

Sequence parallelism: Long sequence training from system perspective
Original
Shenggui Li, Fuzhao Xue, Chaitanya Baranwal, Yongbin Li, and Yang You · 2021
Earlier work this paper cites.
Competition-level code generation with alphacode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals · 2022
Earlier work this paper cites.
Mamba: Linear-time sequence modeling with selective state spaces
Original
Albert Gu and Tri Dao · 2023
Earlier work this paper cites.
Longcoder: A long-range pre-trained language model for code completion, 2023
Original
Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley · 2023
Earlier work this paper cites.
Platypus: Quick, cheap, and powerful refinement of llms
Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz · 2023
Earlier work this paper cites.
Octopack: Instruction tuning code large language models, 2023
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre · 2023
Earlier work this paper cites.
Code llama: Open foundation models for code, 2023
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve · 2023
Earlier work this paper cites.

Similar

Effective long-context scaling of foundation models
Original
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al · 2023
Cited alongside, same era.
Metamath: Bootstrap your own mathematical questions for large language models
Original
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu · 2023
Cited alongside, same era.
” paraphrasing the original text” makes high accuracy long-context qa
Original
Yijiong Yu · 2023
Cited alongside, same era.
Mammoth: Building math generalist models through hybrid instruction tuning
Original
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen · 2023
Cited alongside, same era.

Then

Codegemma: Open code models based on gemma
CodeGemma Team, Ale Jakse Hartman, Andrea Hu, Christopher A. Choquette-Choo, Heri Zhao, Jane Fine, Jeffrey Hui, Jingyue Shen, Joe Kelley, Joshua Howland, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Nam Nguyen, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, Siqi Zuo, Tris Warkentin, and Zhitao et al. Gong · 2024
Closest in time.
Data engineering for scaling language models to 128k context
Original
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng · 2024
Closest in time.
Repoqa: Evaluating long context code understanding
Original
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang · 2024
Closest in time.
Granite code models: A family of open foundation models for code intelligence
Original
Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, et al · 2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Longbench: A bilingual, multitask benchmark for long context understanding, 2024

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li · 2024

Cited alongside, same era.

Api-blend: A comprehensive corpora for training and benchmarking api llms

Original

Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A Lastras · 2024

Cited alongside, same era.

Ring attention with blockwise transformers for near-infinite context, 2023a

Hao Liu, Matei Zaharia, and Pieter Abbeel

Cited in the paper.

Repobench: Benchmarking repository-level code auto-completion systems

Tianyang Liu, Canwen Xu, and Julian McAuley

Cited in the paper.

Repobench: Benchmarking repository-level code auto-completion systems, 2023c

Tianyang Liu, Canwen Xu, and Julian McAuley

Cited in the paper.

How far can camels go? exploring the state of instruction tuning on open resources, 2023a

Original

Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi

Cited in the paper.

Helpsteer: Multi-attribute helpfulness dataset for steerlm, 2023b

Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev

Cited in the paper.

OpenDevin: An Open Platform for AI Software Developers as Generalist Agents

OpenDevin Team · 2024

Closest in time.

Stable code technical report

Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, Ashish Datta, Maksym Zhuravinskyi, Dakota Mahan, Marco Bellagente, Carlos Riquelme, et al · 2024

Closest in time.