2023

An Exploratory Literature Study on Sharing and Energy Use of Language Models for Source Code

Hort, Max, Grishina, Anastasiia, Moonen, Leon

Understand

Large language models trained on source code can support a variety of software development tasks, such as code recommendation and program repair.

Large amounts of data for training such models benefit the models' performance.
However, the size of the data and models results in long training times and high energy consumption.
While publishing source code allows for replicability, users need to repeat the expensive training process if models are not shared.

Built on

“Language Models Are Few-Shot Learners”
Tom. Brown et al · 1901
Earlier work this paper cites.
“A Literature Study of Embeddings on Source Code”
Zimin Chen et al · 1904
Earlier work this paper cites.
“Semantic Source Code Models Using Identifier Embeddings”
Vasiliki Efstathiou et al · 1904
Earlier work this paper cites.
“Quantifying the Carbon Emissions of Machine Learning”
Alexandre Lacoste et al · 1910
Earlier work this paper cites.
“Combining Graph-Based Learning With Automated Data Collection for Code Vulnerability Detection”
H. Wang et al · 1958
Earlier work this paper cites.
“HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing”
Xiyou Zhou et al · 2002
Earlier work this paper cites.
“CodeBERT: A Pre-Trained Model for Programming and Natural Languages”
Zhangyin Feng et al · 2002
Earlier work this paper cites.
“Improved Code Summarization via a Graph Neural Network”
Alexander LeClair et al · 2004
Earlier work this paper cites.
“A Structural Model for Contextual Code Changes”
Shaked Brody et al · 2005
Earlier work this paper cites.
“IntelliCode Compose: Code Generation Using Transformer”
Alexey Svyatkovskiy et al · 2005
Earlier work this paper cites.
“LambdaNet: Probabilistic Type Inference Using Graph Neural Networks”
Jiayi Wei et al · 2005
Earlier work this paper cites.
“Unsupervised Translation of Programming Languages”
Marie-Anne Lachaux et al · 2006
Earlier work this paper cites.
“Neural Code Search Revisited: Enhancing Code Snippet Retrieval through Natural Language Intent”
Geert Heyman et al · 2008
Earlier work this paper cites.
“Deep Learning Based Vulnerability Detection: Are We There Yet?”
Saikat Chakraborty et al · 2009
Earlier work this paper cites.
“GraphCodeBERT: Pre-training Code Representations with Data Flow”
Daya Guo et al · 2009
Earlier work this paper cites.
“Greening through IT: Information Technology for Environmental Sustainability”, 2010
Bill Tomlinson · 2010
Earlier work this paper cites.
“Enhancing Software Engineering Processes towards Sustainable Software Product Design”
Markus Dick et al · 2010
Earlier work this paper cites.
“Deep Just-In-Time Inconsistency Detection Between Comments and Source Code”
Sheena Panthaplackel et al · 2010
Earlier work this paper cites.
“The GREENSOFT Model: A Reference Model for Green and Sustainable Software and Its Engineering”
Stefan Naumann et al · 2011
Earlier work this paper cites.
“Green Cloud Computing: Balancing Energy in Processing, Storage, and Transport”
J Baliga et al · 2011
Earlier work this paper cites.
“The Five Most Neglected Issues in ”Green IT””
Lorenz Hilty et al · 2011
Earlier work this paper cites.
“Systematic Literature Studies: Database Searches vs. Backward Snowballing”
Samireh Jalali et al · 2012
Earlier work this paper cites.
“Distributed Representations of Words and Phrases and Their Compositionality”
Tomas Mikolov et al · 2013
Earlier work this paper cites.
“Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering”
Claes Wohlin · 2014
Earlier work this paper cites.
“A Practical Model for Evaluating the Energy Efficiency of Software Applications”
Georgios Kalaitzoglou et al · 2014
Earlier work this paper cites.
“Framing Sustainability as a Property of Software Quality”
Patricia Lago et al · 2015
Earlier work this paper cites.
“Green Mining: A Methodology of Relating Software Change and Configuration to Power Consumption”
Abram Hindle · 2015
Earlier work this paper cites.
“Guidelines for Conducting Systematic Mapping Studies in Software Engineering: An Update”
Kai Petersen et al · 2015
Earlier work this paper cites.
“Predicting Program Properties from ”Big Code””
Veselin Raychev et al · 2015
Earlier work this paper cites.
“Suggesting Accurate Method and Class Names”
Miltiadis Allamanis et al · 2015
Earlier work this paper cites.
“The FAIR Guiding Principles for Scientific Data Management and Stewardship”
Mark. Wilkinson et al · 2016
Earlier work this paper cites.
“Evaluating the Energy Efficiency of Deep Convolutional Neural Networks on CPUs and GPUs”
Da Li et al · 2016
Earlier work this paper cites.
“A Map of Threats to Validity of Systematic Literature Reviews in Software Engineering”
Xin Zhou et al · 2016
Earlier work this paper cites.
“Identification of Energy Hotspots: A Case Study of the Very Fast Decision Tree”
Eva Garcia-Martin et al · 2017
Earlier work this paper cites.
“A Syntactic Neural Model for General-Purpose Code Generation”
Pengcheng Yin et al · 2017
Earlier work this paper cites.
“A Survey of App Store Analysis for Software Engineering”
William Martin et al · 2017
Earlier work this paper cites.
“A Software Sustainability-Quality Model”, 2018
Nelly Condori-Fernandez et al · 2018
Earlier work this paper cites.
“Empirical Evaluation of the Energy Impact of Refactoring Code Smells”
Roberto Verdecchia et al · 2018
Earlier work this paper cites.
“DeepSim: Deep Learning Code Functional Similarity”
Gang Zhao et al · 2018
Earlier work this paper cites.
“Path-Based Function Embedding and Its Application to Specification Mining”
Daniel DeFreez et al · 2018
Earlier work this paper cites.
“A Retrieve-and-Edit Framework for Predicting Structured Outputs”
Tatsunori. Hashimoto et al · 2018
Earlier work this paper cites.
“Neural Sketch Learning for Conditional Program Generation”
Vijayaraghavan Murali et al · 2018
Earlier work this paper cites.
“StaQC: A Systematically Mined Question-Code Dataset from Stack Overflow”
Ziyu Yao et al · 2018
Earlier work this paper cites.

Similar

“Cross-Project Transfer Representation Learning for Vulnerable Function Discovery”
Guanjun Lin et al · 2018
Cited alongside, same era.
“DeepBugs: A Learning Approach to Name-Based Bug Detection”
Michael Pradel et al · 2018
Cited alongside, same era.
“A General Path-Based Representation for Predicting Program Properties”
Uri Alon et al · 2018
Cited alongside, same era.
“Deep Learning Type Inference”
Vincent. Hellendoorn et al · 2018
Cited alongside, same era.
“SPT-code: Sequence-to-Sequence Pre-Training for Learning Source Code Representations”
Changan Niu et al · 2018
Cited alongside, same era.
“VideoBERT: A Joint Model for Video and Language Representation Learning”
Chen Sun et al · 2019
Cited alongside, same era.
“Energy and Policy Considerations for Deep Learning in NLP”
Emma Strubell et al · 2019
Cited alongside, same era.
“Towards a Software Sustainability-Quality Model: Insights from a Multi-Case Study”
Nelly Condori-Fernandez et al · 2019
Cited alongside, same era.
“Estimation of Energy Consumption in Machine Learning”
Eva García-Martín et al · 2019
Cited alongside, same era.
“Neural Detection of Semantic Code Clones Via Tree-Based Convolution”
Hao Yu et al · 2019
Cited alongside, same era.
“Pythia: AI-assisted Code Completion System”
Alexey Svyatkovskiy et al · 2019
Cited alongside, same era.
“Deep Reinforcement Learning for Syntactic Error Repair in Student Programs”
Rahul Gupta et al · 2019
Cited alongside, same era.

Then

“A Survey on Machine Learning Techniques for Source Code Analysis”
Tushar Sharma et al · 2021
Later among the works it cites.
“CURE: Code-Aware Neural Machine Translation for Automatic Program Repair”
Nan Jiang et al · 2021
Later among the works it cites.
“TFix: Learning to Fix Coding Errors with a Text-to-Text Transformer”
Berkay Berabi et al · 2021
Later among the works it cites.
“CoSQA: 20,000+ Web Queries for Code Search and Question Answering”
Junjie Huang et al · 2021
Later among the works it cites.
“CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Trees”
Ensheng Shi et al · 2021
Later among the works it cites.
“A Multi-Modal Transformer-based Code Summarization Approach for Smart Contracts”
Zhen Yang et al · 2021
Later among the works it cites.
“CodeQA: A Question Answering Dataset for Source Code Comprehension”
Chenxiao Liu et al · 2021
Later among the works it cites.
“Vulnerability Detection with Fine-Grained Interpretations”
Yi Li et al · 2021
Later among the works it cites.
“DeepWukong: Statically Detecting Software Vulnerabilities Using Deep Graph Neural Network”
Xiao Cheng et al · 2021
Later among the works it cites.
“ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generation”
Weizhen Qi et al · 2021
Later among the works it cites.
“DOBF: A Deobfuscation Pre-Training Objective for Programming Languages”
Baptiste Roziere et al · 2021
Later among the works it cites.
“CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”
Yue Wang et al · 2021
Later among the works it cites.
“Unified Pre-training for Program Understanding and Generation”
Wasi Ahmad et al · 2021
Later among the works it cites.
“Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks”
Antonio Mastropaolo et al · 2021
Later among the works it cites.
“CodeTrans: Towards Cracking the Language of Silicon’s Code Through Self-Supervised Deep Learning and High Performance Computing”
Ahmed Elnaggar et al · 2021
Later among the works it cites.
“JavaBERT: Training a Transformer-Based Model for the Java Programming Language”
Nelson de Sousa et al · 2021
Later among the works it cites.
“CoTexT: Multi-task Learning with Code-Text Transformer”
Long Phan et al · 2021
Later among the works it cites.
“CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation”
Shuai Lu et al · 2021
Later among the works it cites.
“Contrastive Code Representation Learning”
Paras Jain et al · 2021
Later among the works it cites.
“Language-Agnostic Representation Learning of Source Code from Structure and Context”
Daniel Zügner et al · 2021
Later among the works it cites.
“Green Algorithms: Quantifying the Carbon Footprint of Computation”
Loïc Lannelongue et al · 2021
Later among the works it cites.
“On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?”
Emily. Bender et al · 2021
Later among the works it cites.
“Green AI: Do Deep Learning Frameworks Have Different Costs?”
Stefanos Georgiou et al · 2022
Later among the works it cites.
“StructCoder: Structure-Aware Transformer for Code Generation”
Sindhu Tipirneni et al · 2022
Later among the works it cites.
“BLOOM: A 176B-Parameter Open-Access Multilingual Language Model”
Teven Le et al · 2022
Later among the works it cites.
“Energy Consumption of Automated Program Repair”
Matias Martinez et al · 2022
Later among the works it cites.
“Data-Centric Green AI: An Exploratory Empirical Study”
Roberto Verdecchia et al · 2022
Later among the works it cites.
“Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning”
Peter Henderson et al · 2022
Later among the works it cites.
“Analysing the Energy Impact of Different Optimisations for Machine Learning Models”
Maria Gutierrez et al · 2022
Later among the works it cites.
“A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research”
Cody Watson et al · 2022
Later among the works it cites.
“Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code”
Changan Niu et al · 2022
Later among the works it cites.
“CODIT: Code Editing With Tree-Based Neural Models”
Saikat Chakraborty et al · 2022
Later among the works it cites.
“Neural Program Repair with Execution-Based Backpropagation”
He Ye et al · 2022
Later among the works it cites.
“VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection”
Hazim Hanif et al · 2022
Later among the works it cites.
“Type4Py: Practical Deep Similarity Learning-Based Type Inference for Python”
Amir. Mir et al · 2022
Later among the works it cites.
“Global Relational Models of Source Code”
Vincent. Hellendoorn et al · 2022
Later among the works it cites.
“GraphCode2Vec: Generic Code Embedding via Lexical and Program Dependence Analyses”
Wei Ma et al · 2022
Later among the works it cites.
“Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model”
Alexandra Luccioni et al · 2022
Later among the works it cites.
“A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model”
Imad Lakim et al · 2022
Later among the works it cites.
“Improving Reproducibility in Machine Learning Research (a Report from the NeurIPS 2019 Reproducibility Program)”
Joelle Pineau et al · 2022
Later among the works it cites.
“A Systematic Review of Green AI”
Roberto Verdecchia et al · 2023
Closest in time.
“Neural Transfer Learning for Repairing Security Vulnerabilities in C Code”
Zimin Chen et al · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…