2023

CodeFuse-13B: A Pretrained Multi-lingual Code Large Language Model

Di, Peng, Li, Jianguo, Yu, Hang et al.

Understand

Code Large Language Models (Code LLMs) have gained significant attention in the industry due to their wide applications in the full lifecycle of software engineering.

However, the effectiveness of existing models in understanding non-English inputs for multi-lingual code-related tasks is still far from well studied.
This paper introduces CodeFuse-13B, an open-sourced pre-trained code LLM.
It is specifically designed for code-related tasks with both English and Chinese prompts and supports over 40 programming languages.

Built on

ERNIE: Enhanced Representation through Knowledge Integration
Original
Yu Sun, Shuohuan Wang, Yukun Li, et al · 1904
Earlier work this paper cites.
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
Original
Yu Sun, Shuohuan Wang, Yukun Li, et al · 1907
Earlier work this paper cites.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Original
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, et al · 1909
Earlier work this paper cites.
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
Original
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020 · 1910
Earlier work this paper cites.
Byte Pair Encoding: A Text Compression Scheme That Accelerates Pattern Matching
Yusuke Shibata, Takuya Kida, Shuichi Fukamachi, et al · 1999
Earlier work this paper cites.
On Layer Normalization in the Transformer Architecture
Original
Ruibin Xiong, Yunchang Yang, Di He, et al · 2002
Earlier work this paper cites.
Language Models are Few-Shot Learners
Original
Tom B. Brown, Benjamin Mann, Nick Ryder, et al · 2005
Earlier work this paper cites.
Keynote Address: .QL for Source Code Analysis. In Seventh IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2007) . 3–16
Oege de Moor, Mathieu Verbaere, Elnar Hajiyev, et al · 2007
Earlier work this paper cites.
Layer Normalization
Original
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016 · 2016
Earlier work this paper cites.
Adam: A Method for Stochastic Optimization
Original
Diederik P. Kingma and Jimmy Ba. 2017 · 2017
Earlier work this paper cites.
Pinpoint: fast and precise sparse value flow analysis for million lines of code. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2018, Philadelphia, PA, USA, June 18-22, 2018 , Jeffrey S. Foster and Dan Grossman (Eds.). ACM, 693–706
Qingkai Shi, Xiao Xiao, Rongxin Wu, et al · 2018
Earlier work this paper cites.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Earlier work this paper cites.
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Original
Leo Gao, Stella Biderman, Sid Black, et al · 2020
Earlier work this paper cites.
Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations . Association for Computational Linguistics, Online, 38–45
Thomas Wolf, Lysandre Debut, Victor Sanh, et al · 2020
Earlier work this paper cites.
Program Synthesis with Large Language Models
Original
Jacob Austin, Augustus Odena, Maxwell Nye, et al · 2021
Earlier work this paper cites.
GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model
Ben Wang and Aran Komatsuzaki. 2021 · 2021
Earlier work this paper cites.

Similar

Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzing. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI 2021) . Association for Computing Machinery, 435–450
Guixin Ye, Zhanyong Tang, Shin Hwei Tan, Songfang Huang, et al · 2021
Cited alongside, same era.
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Original
Sid Black, Stella Biderman, Eric Hallahan, et al · 2022
Cited alongside, same era.
PaLM: Scaling Language Modeling with Pathways
Original
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, et al · 2022
Cited alongside, same era.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Original
Tri Dao, Daniel Y. Fu, Stefano Ermon, et al · 2022
Cited alongside, same era.
GPTQ: Accurate post-training quantization for generative pre-trained transformers
Original
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2022 · 2022
Cited alongside, same era.
Training Compute-Optimal Large Language Models
Original
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al · 2022
Cited alongside, same era.
The Stack: 3 TB of permissively licensed source code
Original
Denis Kocetkov, Raymond Li, Loubna Ben Allal, et al · 2022
Cited alongside, same era.
Competition-level code generation with AlphaCode
Yujia Li, David Choi, Junyoung Chung, et al · 2022
Cited alongside, same era.
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Original
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, et al · 2022
Cited alongside, same era.
RoFormer: Enhanced Transformer with Rotary Position Embedding
Original
Jianlin Su, Yu Lu, Shengfeng Pan, et al · 2022
Cited alongside, same era.
GLM-130B: An Open Bilingual Pre-trained Model
Original
Aohan Zeng, Xiao Liu, Zhengxiao Du, et al · 2022
Cited alongside, same era.
Field-Based Static Taint Analysis for Industrial Microservices. In 44th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2022, Pittsburgh, PA, USA, May 22-24, 2022 . IEEE, 149–150
Zexin Zhong, Jiangchao Liu, Diyu Wu, Peng Di, et al · 2022
Cited alongside, same era.

Then

Gaussian Error Linear Units (GELUs)
Original
Dan Hendrycks and Kevin Gimpel. 2023 · 2023
Closest in time.
StarCoder: may the source be with you!
Original
Raymond Li, Loubna Ben Allal, Yangtian Zi, et al · 2023
Closest in time.
Hybrid Inlining: A Framework for Compositional and Context-Sensitive Static Analysis. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, Seattle, WA, USA, July 17-21, 2023 , René Just and Gordon Fraser (Eds.). ACM, 114–126
Jiangchao Liu, Jierui Liu, Peng Di, et al · 2023
Closest in time.
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Original
Ziyang Luo, Can Xu, Pu Zhao, et al · 2023
Closest in time.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Original
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, et al · 2023
Closest in time.
GPT-4 Technical Report
Original
OpenAI. 2023 · 2023
Closest in time.
Code Llama: Open Foundation Models for Code
Original
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, et al · 2023
Closest in time.
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
Original
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023 · 2023
Closest in time.
PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedback
Original
Bo Shen, Jiaxin Zhang, Taihong Chen, et al · 2023
Closest in time.
Stanford Alpaca: An Instruction-following LLaMA model
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, et al · 2023
Closest in time.
Llama 2: Open foundation and fine-tuned chat models
Original
Hugo Touvron, Louis Martin, Kevin Stone, et al · 2023
Closest in time.
A Survey of Large Language Models
Original
Wayne Xin Zhao, Kun Zhou, Junyi Li, et al · 2023
Closest in time.
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X
Original
Qinkai Zheng, Xiao Xia, Xu Zou, et al · 2023
Closest in time.
Scalable Compositional Static Taint Analysis for Sensitive Data Tracing on Industrial Micro-Services. In 45th IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice, SEIP@ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 110–121
Zexin Zhong, Jiangchao Liu, Diyu Wu, Peng Di, et al · 2023
Closest in time.
The Devil is in the Tails: How Long-Tailed Code Distributions Impact Large Language Models
Original
Xin Zhou, Kisub Kim, Bowen Xu, et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…