On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generation

Binary codes capable of correcting deletions, insertions and reversals

Vladimir Iosifovich Levenshtein. 1966 · 1965

Earlier work this paper cites.

Bleu: a method for automatic evaluation of machine translation

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002

Earlier work this paper cites.

ROUGE: A package for automatic evaluation of summaries

Chin-Yew Lin. 2004 · 2004

Earlier work this paper cites.

METEOR: An automatic metric for MT evaluation with improved correlation with human judgments

Satanjeev Banerjee and Alon Lavie. 2005 · 2005

Earlier work this paper cites.

Graphcodebert: Pre-training code representations with data flow

Original

Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020 · 2009

Earlier work this paper cites.

Codebleu: a method for automatic evaluation of code synthesis

Original

Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2009

Earlier work this paper cites.

Metrics, statistics, tests

Tetsuya Sakai. 2014 · 2013

Earlier work this paper cites.

When to use the bonferroni correction

Richard A Armstrong. 2014 · 2014

Earlier work this paper cites.

chrF: character n-gram F-score for automatic MT evaluation

Maja Popović. 2015 · 2015

Earlier work this paper cites.

Convolutional neural networks over tree structures for programming language processing

Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016 · 2016

Earlier work this paper cites.

Neural code comprehension: A learnable representation of code semantics

Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018 · 2018

Earlier work this paper cites.

Learning to mine aligned code and natural language pairs from stack overflow

Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018 · 2018

Earlier work this paper cites.

JuICe: A large scale distantly supervised dataset for open domain context-based code generation

Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019 · 2019

Earlier work this paper cites.

Program synthesis and semantic parsing with learned code idioms

Richard Shin, Miltiadis Allamanis, Marc Brockschmidt, and Oleksandr Polozov. 2019 · 2019

Earlier work this paper cites.

A grammar-based structural cnn decoder for code generation

Zeyu Sun, Qihao Zhu, Lili Mou, Yingfei Xiong, Ge Li, and Lu Zhang. 2019 · 2019

Earlier work this paper cites.

Does bleu score work for code migration?

Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. 2019 · 2019

Earlier work this paper cites.

A novel neural source code representation based on abstract syntax tree

Jian Zhang, Xu Wang, Hongyu Zhang, Hailong Sun, Kaixuan Wang, and Xudong Liu. 2019a · 2019

Earlier work this paper cites.

CodeBERT: A pre-trained model for programming and natural languages

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020 · 2020

Earlier work this paper cites.

COMET: A neural framework for MT evaluation

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020 · 2020

Earlier work this paper cites.

Unified pre-training for program understanding and generation

WU Ahmad, S Chakraborty, B Ray, and KW Chang. 2021 · 2021

Earlier work this paper cites.

Program synthesis with large language models

Original

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021 · 2021

Earlier work this paper cites.

Evaluating large language models trained on code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021 · 2021

Earlier work this paper cites.

Measuring coding challenge competence with apps

Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021 · 2021

Earlier work this paper cites.

Multipl-e: A scalable and extensible approach to benchmarking neural code generation

Original

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022 · 2022

Earlier work this paper cites.

Incoder: A generative model for code infilling and synthesis

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2022 · 2022

Earlier work this paper cites.

Unixcoder: Unified cross-modal pre-training for code representation

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022 · 2022

Cited alongside, same era.

Execution-based evaluation for data science code generation models

Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022 · 2022

Cited alongside, same era.

Probing semantic grounding in language models of code with representational similarity analysis

Original

Shounak Naik, Rajaswa Patil, Swati Agarwal, and Veeky Baths. 2022 · 2022

Cited alongside, same era.

Codegen: An open large language model for code with multi-turn program synthesis

Original

Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022 · 2022

Cited alongside, same era.

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022 · 2022

Progprompt: Generating situated robot task plans using large language models

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2023 · 2023

Later among the works it cites.

Llama 2: Open foundation and fine-tuned chat models

Original

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 · 2023

Later among the works it cites.

Codet5+: Open code large language models for code understanding and generation

Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023a · 2023

Later among the works it cites.

Mconala: A benchmark for code generation from multiple natural languages

Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F Xu, and Graham Neubig. 2023b · 2023

Later among the works it cites.

Magicoder: Source code is all you need

Original

Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023 · 2023