2024

SparseCoder: Identifier-Aware Sparse Transformer for File-Level Code Summarization

Wang, Yanlin, Huang, Yanxian, Guo, Daya et al.

Understand

Code summarization aims to generate natural language descriptions of source code, facilitating programmers to understand and maintain it rapidly.

While previous code summarization efforts have predominantly focused on method-level, this paper studies file-level code summarization, which can assist programmers in understanding and maintaining large source code projects.
Unlike method-level code summarization,file-level code summarization typically involves long source code within a single file, which makes it challenging for Transformer-based models to understand the code semantics for the maximum input length of these models is difficult to set to a large number that can handle long code input well, due to the quadratic scaling of computational complexity with the input sequence length.
To address this challenge, we propose SparseCoder, an identifier-aware sparse transformer for effectively handling long code sequences.

Built on

“CodeBERT: A Pre-Trained Model for Programming and Natural Languages”
Original
Zhangyin Feng et al · 2002
Earlier work this paper cites.
“ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation”
Chin-Yew Lin and Franz Och · 2004
Earlier work this paper cites.
“Supporting program comprehension with source code summarization”
Sonia Haiduc, Jairo Aponte and Andrian Marcus · 2010
Earlier work this paper cites.
“On the Use of Automated Text Summarization Techniques for Summarizing Source Code”
Sonia Haiduc, Jairo Aponte, Laura Moreno and Andrian Marcus · 2010
Earlier work this paper cites.
“Towards automatically generating summary comments for Java methods”
Giriprasad Sridhara et al · 2010
Earlier work this paper cites.
“Evaluating source code summarization techniques: Replication and expansion”
Brian. Eddy, Jeffrey. Robinson, Nicholas. Kraft and Jeffrey. Carver · 2013
Earlier work this paper cites.
“Natural Language Models for Predicting Programming Comments”
Dana Movshovitz-Attias and William. Cohen · 2013
Earlier work this paper cites.
“Automatic documentation generation via source code summarization of method context”
Paul McBurney and Collin McMillan · 2014
Earlier work this paper cites.
“Improving automated source code summarization via an eye-tracking study of programmers”
Paige Rodeghero et al · 2014
Earlier work this paper cites.
“Towards a big data curated benchmark of inter-project code clones”
Jeffrey Svajlenko et al · 2014
Earlier work this paper cites.
“Automatic comment generation using a neural translation model”, 2016
Tjalling Haije · 2016
Earlier work this paper cites.
“Summarizing Source Code using a Neural Attention Model”
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung and Luke Zettlemoyer · 2016
Earlier work this paper cites.
“Attention is all you need”
Ashish Vaswani et al · 2017
Earlier work this paper cites.
“Bert: Pre-training of deep bidirectional transformers for language understanding”
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2018
Earlier work this paper cites.
“Deep code comment generation”
Xing Hu et al · 2018
Earlier work this paper cites.
“Summarizing Source Code with Transferred API Knowledge”
Xing Hu et al · 2018
Earlier work this paper cites.
“Improving automatic source code summarization via deep reinforcement learning”
Yao Wan et al · 2018
Earlier work this paper cites.

Similar

“The adverse effects of code duplication in machine learning models of code”
Miltiadis Allamanis · 2019
Cited alongside, same era.
“Generating long sequences with sparse transformers”
Original
Rewon Child, Scott Gray, Alec Radford and Ilya Sutskever · 2019
Cited alongside, same era.
“Transformer-xl: Attentive language models beyond a fixed-length context”
Zihang Dai et al · 2019
Cited alongside, same era.
“Deep code comment generation with hybrid lexical and syntactical information”
Xing Hu et al · 2019
Cited alongside, same era.
“A neural model for generating natural language summaries of program subroutines”
Alexander LeClair, Siyuan Jiang and Collin McMillan · 2019
Cited alongside, same era.
“Automatic generation of pull request descriptions”
Zhongxin Liu et al · 2019
Cited alongside, same era.
“A Transformer-based Approach for Source Code Summarization”
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray and Kai-Wei Chang · 2020
Cited alongside, same era.
“Longformer: The long-document transformer”
Original
Iz Beltagy, Matthew Peters and Arman Cohan · 2020
Cited alongside, same era.
“CodeBERT: A Pre-Trained Model for Programming and Natural Languages”
Zhangyin Feng et al · 2020
Cited alongside, same era.
“Deep code comment generation with hybrid lexical and syntactical information”
Xing Hu et al · 2020
Cited alongside, same era.
“Learning and evaluating contextual embedding of source code”
Aditya Kanade, Petros Maniatis, Gogul Balakrishnan and Kensen Shi · 2020
Cited alongside, same era.
“Improved code summarization via a graph neural network”
Alexander LeClair, Sakib Haque, Linfgei Wu and Collin McMillan · 2020
Cited alongside, same era.

Then

“Graphcodebert: Pre-training code representations with data flow”
Daya Guo et al · 2021
Later among the works it cites.
“Cast: Enhancing code summarization with hierarchical splitting and reconstruction of abstract syntax trees”
Original
Ensheng Shi et al · 2021
Later among the works it cites.
“CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation”
Original
Yue Wang, Weishi Wang, Shafiq. Joty and Steven.. Hoi · 2021
Later among the works it cites.
“Hi-Transformer: hierarchical interactive transformer for efficient and effective long document modeling”
Chuhan Wu, Fangzhao Wu, Tao Qi and Yongfeng Huang · 2021
Later among the works it cites.
“GT-SimNet: Improving code automatic summarization via multi-modal similarity networks”
Xuejian Gao et al · 2022
Later among the works it cites.
“M2TS: Multi-Scale Multi-Modal Approach Based on Transformer for Source Code Summarization”
Yuexiu Gao and Chen Lyu · 2022
Later among the works it cites.
“Source Code Summarization with Structural Relative Position Guided Transformer”
Ziyan Gong et al · 2022
Later among the works it cites.
“UniXcoder: Unified Cross-Modal Pre-training for Code Representation”
Daya Guo et al · 2022
Later among the works it cites.
“Modeling Hierarchical Syntax Structure with Triplet Position for Source Code Summarization”
Juncai Guo et al · 2022
Later among the works it cites.
“LongT5: Efficient Text-To-Text Transformer for Long Sequences”
Mandy Guo et al · 2022
Later among the works it cites.
“LoRA: Low-Rank Adaptation of Large Language Models”
Edward Hu et al · 2022
Later among the works it cites.
“Understanding Long Programming Languages with Structure-Aware Sparse Attention”
Tingting Liu et al · 2022
Later among the works it cites.
“Boosting Code Summarization by Embedding Code Structures”
Jikyoeng Son, Joonghyuk Hahn, HyeonTae Seo and Yo-Sub Han · 2022
Later among the works it cites.
“GypSum: Learning Hybrid Representations for Code Summarization”
Yu Wang, Yu Dong, Xuesong Lu and Aoying Zhou · 2022
Later among the works it cites.
“LSG Attention: Extrapolation of pretrained Transformers to long sequences”
Charles Condevaux and Sébastien Harispe · 2023
Later among the works it cites.
“Summarizing source code using a neural attention model”
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung and Luke Zettlemoyer · 2083
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…