2021

On the Evaluation of Neural Code Summarization

Shi, Ensheng, Wang, Yanlin, Du, Lun et al.

Understand

Source code summaries are important for program comprehension and maintenance.

However, there are plenty of programs with missing, outdated, or mismatched summaries.
Recently, deep learning techniques have been exploited to automatically generate summaries for given code snippets.
To achieve a profound understanding of how far we are from solving this problem and provide suggestions to future research, in this paper, we conduct a systematic and in-depth analysis of 5 state-of-the-art neural code summarization models on 6 widely used BLEU variants, 4 pre-processing operations and their combinations, and 3 widely used datasets.

Built on

CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 1909
Earlier work this paper cites.
The treatment of ties in ranking problems
Maurice G Kendall. 1945 · 1945
Earlier work this paper cites.
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
H. B. Mann and D. R. Whitney. 1947 · 1947
Earlier work this paper cites.
Documenting software systems with views. In SIGDOC . ACM, 211–219
Scott R. Tilley, Hausi A. Müller, and Mehmet A. Orgun. 1992 · 1992
Earlier work this paper cites.
Standard probability and statistics tables and formulae
Eric R Ziegel. 2001 · 2001
Earlier work this paper cites.
Modelling out-of-vocabulary words for robust speech recognition
Issam Bazzi. 2002 · 2002
Earlier work this paper cites.
The relevance of software documentation, tools and technologies: a survey. In ACM Symposium on Document Engineering . ACM, 26–33
Andrew Forward and Timothy Lethbridge. 2002 · 2002
Earlier work this paper cites.
Bleu: a Method for Automatic Evaluation of Machine Translation. In ACL . ACL, 311–318
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Software Documentation: How Much Is Enough?. In CSMR . IEEE Computer Society, 13
Lionel C. Briand. 2003 · 2003
Earlier work this paper cites.
ROUGE: A Package for Automatic Evaluation of Summaries. In ACL
Chin-Yew Lin. 2004 · 2004
Earlier work this paper cites.
METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In IEEvaluation@ACL
Satanjeev Banerjee and Alon Lavie. 2005 · 2005
Earlier work this paper cites.
AutoML: Evaluating models
Google Cloud. 2007 · 2007
Earlier work this paper cites.
Answering the call for a standard reliability measure for coding data
Andrew F Hayes and Klaus Krippendorff. 2007 · 2007
Earlier work this paper cites.
Large-scale code reuse in open source software. In First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS’07: ICSE Workshops 2007) . IEEE, 7–7
Audris Mockus. 2007 · 2007
Earlier work this paper cites.
Towards automatically generating summary comments for Java methods. In ASE . ACM, 43–52
Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori L. Pollock, and K. Vijay-Shanker. 2010 · 2010
Earlier work this paper cites.
Statistics for research . Vol. 512
Shirley Dowdy, Stanley Wearden, and Daniel Chilko. 2011 · 2011
Earlier work this paper cites.
Evaluating source code summarization techniques: Replication and expansion. In ICPC . IEEE Computer Society, 13–22
Brian P. Eddy, Jeffrey A. Robinson, Nicholas A. Kraft, and Jeffrey C. Carver. 2013 · 2013
Earlier work this paper cites.
Better Word Representations with Recursive Neural Networks for Morphology. In CoNLL . ACL, 104–113
Thang Luong, Richard Socher, and Christopher D. Manning. 2013 · 2013
Earlier work this paper cites.
Automatic generation of natural language summaries for Java classes. In ICPC . IEEE Computer Society, 23–32
Laura Moreno, Jairo Aponte, Giriprasad Sridhara, Andrian Marcus, Lori L. Pollock, and K. Vijay-Shanker. 2013 · 2013
Earlier work this paper cites.
A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU. In WMT@ACL . The Association for Computer Linguistics, 362–367
Boxing Chen and Colin Cherry. 2014 · 2014
Earlier work this paper cites.
Improving automated source code summarization via an eye-tracking study of programmers. In ICSE . ACM, 390–401
Paige Rodeghero, Collin McMillan, Paul W. McBurney, Nigel Bosch, and Sidney K. D’Mello. 2014 · 2014
Earlier work this paper cites.
CIDEr: Consensus-based image description evaluation. In CVPR
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015 · 2015
Earlier work this paper cites.
Summarizing Source Code using a Neural Attention Model. In ACL (1) . The Association for Computer Linguistics
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016 · 2016
Earlier work this paper cites.

Similar

Summarizing software artifacts: A literature review
Najam Nazar, Yan Hu, and He Jiang. 2016 · 2016
Cited alongside, same era.
Improving Neural Language Models with a Continuous Cache. In ICLR (Poster) . OpenReview.net
Edouard Grave, Armand Joulin, and Nicolas Usunier. 2017 · 2017
Cited alongside, same era.
Are deep neural networks the best choice for modeling source code?. In ESEC/SIGSOFT FSE . ACM, 763–773
Vincent J. Hellendoorn and Premkumar T. Devanbu. 2017 · 2017
Cited alongside, same era.
DéjàVu: a map of code duplicates on GitHub. In OOPSLA
Cristina V Lopes, Petr Maj, Pedro Martins, Vaibhav Saini, Di Yang, Jakub Zitny, Hitesh Sajnani, and Jan Vitek. 2017 · 2017
Cited alongside, same era.
Pointer Sentinel Mixture Models. In ICLR (Poster) . OpenReview.net
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017 · 2017
Cited alongside, same era.
Get To The Point: Summarization with Pointer-Generator Networks. In ACL (1) . Association for Computational Linguistics, 1073–1083
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017 · 2017
Cited alongside, same era.
A neural framework for retrieval and summarization of source code. In ASE . ACM, 826–831
Qingying Chen and Minghui Zhou. 2018 · 2018
Cited alongside, same era.
A Call for Clarity in Reporting BLEU Scores. In WMT . Association for Computational Linguistics, 186–191
Matt Post. 2018 · 2018
Cited alongside, same era.
Improving automatic source code summarization via deep reinforcement learning. In ASE . ACM, 397–407
Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018 · 2018
Cited alongside, same era.
The adverse effects of code duplication in machine learning models of code. In Onward! ACM, 143–153
Miltiadis Allamanis. 2019 · 2019
Cited alongside, same era.
code2seq: Generating Sequences from Structured Representations of Code. In ICLR (Poster) . OpenReview.net
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. 2019 · 2019
Cited alongside, same era.
Structured Neural Summarization. In ICLR
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 2019
Cited alongside, same era.

Then

Code to Comment "Translation": Data, Metrics, Baselining & Evaluation. In ASE . IEEE, 746–757
David Gros, Hariharan Sezhiyan, Prem Devanbu, and Zhou Yu. 2020 · 2020
Later among the works it cites.
Improved automatic summarization of subroutines via attention to file context. In MSR
Sakib Haque, Alexander LeClair, Lingfei Wu, and Collin McMillan. 2020 · 2020
Later among the works it cites.
Deep code comment generation with hybrid lexical and syntactical information
Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2020 · 2020
Later among the works it cites.
Big code != big vocabulary: open-vocabulary models for sfBLEUzource code. In ICSE . ACM, 1073–1085
Rafael-Michael Karampatsis, Hlib Babii, Romain Robbes, Charles Sutton, and Andrea Janes. 2020 · 2020
Later among the works it cites.
Improved Code Summarization via a Graph Neural Network. In ICPC . ACM, 184–195
Alexander LeClair, Sakib Haque, Lingfei Wu, and Collin McMillan. 2020 · 2020
Later among the works it cites.
Learning to Update Natural Language Comments Based on Code Changes. In ACL . Association for Computational Linguistics, 1853–1868
Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond J. Mooney. 2020 · 2020
Later among the works it cites.
A Human Study of Comprehension and Code Summarization. In ICPC . ACM, 2–13
Sean Stapleton, Yashmeet Gambhir, Alexander LeClair, Zachary Eberhart, Westley Weimer, Kevin Leach, and Yu Huang. 2020 · 2020
Later among the works it cites.
Reinforcement-learning-guided source code summarization via hierarchical attention
Wenhua Wang, Yuqun Zhang, Yulei Sui, Yao Wan, Zhou Zhao, Jian Wu, Philip Yu, and Guandong Xu. 2020 · 2020
Later among the works it cites.
Retrieve and Refine: Exemplar-based Neural Comment Generation. In ASE . IEEE, 349–360
Bolin Wei, Yongmin Li, Ge Li, Xin Xia, and Zhi Jin. 2020 · 2020
Later among the works it cites.
Leveraging code generation to improve code retrieval and summarization via dual learning. In The Web Conference
Wei Ye, Rui Xie, Jinglei Zhang, Tianxiang Hu, Xiaoyin Wang, and Shikun Zhang. 2020 · 2020
Later among the works it cites.
Retrieval-based neural source code summarization. In ICSE . ACM, 1385–1397
Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, and Xudong Liu. 2020 · 2020
Later among the works it cites.
Project-Level Encoding for Neural Source Code Summarization of Subroutines. In ICPC
Aakash Bansal, Sakib Haque, and Collin McMillan. 2021 · 2021
Closest in time.
Ensemble Models for Neural Source Code Summarization of Subroutines
Original
Alexander LeClair, Aakash Bansal, and Collin McMillan. 2021 · 2021
Closest in time.
Improving Code Summarization with Block-wise Abstract Syntax Tree Splitting. In ICPC
Chen Lin, Zhichao Ouyang, Junqing Zhuang, Jianqiang Chen, Hui Li, and Rongxin Wu. 2021 · 2021
Closest in time.
Deep Just-In-Time Inconsistency Detection Between Comments and Source Code. In AAAI . AAAI Press, 427–435
Sheena Panthaplackel, Junyi Jessy Li, Milos Gligoric, and Raymond J. Mooney. 2021 · 2021
Closest in time.
Reassessing automatic evaluation metrics for code summarization tasks. In ESEC/SIGSOFT FSE . ACM, 1105–1116
Devjeet Roy, Sarah Fakhoury, and Venera Arnaoudova. 2021 · 2021
Closest in time.
CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Trees. In EMNLP (1) . Association for Computational Linguistics, 4053–4062
Ensheng Shi, Yanlin Wang, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Hongbin Sun. 2021 · 2021
Closest in time.
On the Evaluation of Commit Message Generation Models: An Experimental Study
Original
Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021 · 2021
Closest in time.
CoCoSum: Contextual Code Summarization with Multi-Relational Graph Neural Network
Original
Yanlin Wang, Ensheng Shi, Lun Du, Xiaodi Yang, Yuxuan Hu, Shi Han, Hongyu Zhang, and Dongmei Zhang. 2021 · 2021
Closest in time.
Code Summarization with Structure-induced Transformer. In ACL/IJCNLP (Findings) . Association for Computational Linguistics, 1078–1090
Hongqiu Wu, Hai Zhao, and Min Zhang. 2021 · 2021
Closest in time.
Exploiting Method Names to Improve Code Summarization: A Deliberation Multi-Task Learning Approach. In ICPC
Rui Xie, Wei Ye, Jinan Sun, and Shikun Zhang. 2021 · 2021
Closest in time.
A Convolutional Attention Network for Extreme Summarization of Source Code. In ICML (JMLR Workshop and Conference Proceedings, Vol. 48) . JMLR.org, 2091–2100
Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016 · 2091
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…