Calibration of Large Language Models on Code Summarization

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901

Earlier work this paper cites.

Verification of forecasts expressed in terms of probability

Glenn W Brier. 1950 · 1950

Earlier work this paper cites.

On the Combination of Forecast Probabilities for Consecutive Precipitation Periods. In Weather and Forecasting , Vol. 5. 640––650

Daniel S. Wilks. 1990 · 1990

Earlier work this paper cites.

Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods

John Platt et al · 1999

Earlier work this paper cites.

Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml , Vol. 1. 609–616

Bianca Zadrozny and Charles Elkan. 2001 · 2001

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining . 694–699

Bianca Zadrozny and Charles Elkan. 2002 · 2002

Earlier work this paper cites.

Rouge: A package for automatic evaluation of summaries. In Text summarization branches out . 74–81

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization . 65–72

Satanjeev Banerjee and Alon Lavie. 2005 · 2005

Earlier work this paper cites.

Do Code and Comments Co-Evolve? On the Relation between Source Code and Comment Changes. In 14th Working Conference on Reverse Engineering (WCRE 2007) . 70–79

Beat Fluri, Michael Wursch, and Harald C. Gall. 2007 · 2007

Earlier work this paper cites.

The probabilistic relevance framework: BM25 and beyond

Stephen Robertson, Hugo Zaragoza, et al · 2009

Earlier work this paper cites.

Towards automatically generating summary comments for java methods. In Proceedings of the 25th IEEE/ACM international conference on Automated software engineering . 43–52

Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock, and K Vijay-Shanker. 2010 · 2010

Earlier work this paper cites.

Nonparametric Statistical Methods (3rd ed.)

Myles Hollander, Douglas A. Wolfe, and Eric Chicken. 2013 · 2013

Earlier work this paper cites.

Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence , Vol. 29

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015 · 2015

Earlier work this paper cites.

An Introduction to Statistical Methods and Data Analysis

R.L. Ott and M.T. Longnecker. 2015 · 2015

Earlier work this paper cites.

On calibration of modern neural networks. In International conference on machine learning . PMLR, 1321–1330

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017 · 2017

Earlier work this paper cites.

Analyzing Uncertainty in Neural Machine Translation

Original

Myle Ott, Michael Auli, David Grangier, and Marc’Aurelio Ranzato. 2018 · 2018

Earlier work this paper cites.

Automatic code summarization: A systematic literature review

Original

Yuxiang Zhu and Minxue Pan. 2019 · 2019

Earlier work this paper cites.

CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020 . 1536–1547

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al · 2020

Cited alongside, same era.

A human study of comprehension and code summarization. In Proceedings of the 28th International Conference on Program Comprehension . 2–13

Sean Stapleton, Yashmeet Gambhir, Alexander LeClair, Zachary Eberhart, Westley Weimer, Kevin Leach, and Yu Huang. 2020 · 2020

Cited alongside, same era.

On the Inference Calibration of Neural Machine Translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, Online, 3070–3079

Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020 · 2020

Cited alongside, same era.

Unified pre-training for program understanding and generation

Original

Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021 · 2021

Retrieval-based prompt selection for code-related few-shot learning. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2450–2462

Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023 · 2023

Later among the works it cites.

Code llama: Open foundation models for code

Original

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al · 2023

Later among the works it cites.

scipy.stats.false_discovery_control

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2023 · 2023

Later among the works it cites.

PACE: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysis

Original

Dylan Zhang, Xuchao Zhang, Chetan Bansal, Pedro Las-Casas, Rodrigo Fonseca, and Saravan Rajmohan. 2023 · 2023