Fetching the paper…

Reading the bibliography…

2023

Large Language Model-Aware In-Context Learning for Code Generation

Li, Jia, Li, Ge, Tao, Chongyang et al.

Original
PDF
Post on X

Understand

Large language models (LLMs) have shown impressive in-context learning (ICL) ability in code generation.

LLMs take a prompt consisting of requirement-code examples and a new requirement as input, and output new programs.
Existing studies have found that ICL is highly dominated by the examples and thus arises research on example selection.
However, existing approaches randomly select examples or only consider the textual similarity of requirements to retrieve, leading to sub-optimal performance.

Large Language Model-Aware In-Context Learning for Code Generation · Around

Built on

Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 · 2002
Earlier work this paper cites.
Large-scale code reuse in open source software. In First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS’07: ICSE Workshops 2007) . IEEE, 7–7
Audris Mockus. 2007 · 2007
Earlier work this paper cites.
The probabilistic relevance framework: BM25 and beyond
Stephen Robertson, Hugo Zaragoza, et al · 2009
Earlier work this paper cites.
Language to Logical Form with Neural Attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 33–43
Li Dong and Mirella Lapata. 2016 · 2016
Earlier work this paper cites.
A survey of machine learning for big code and naturalness
Miltiadis Allamanis, Earl T Barr, Premkumar Devanbu, and Charles Sutton. 2018a · 2018
Earlier work this paper cites.
A Survey of Machine Learning for Big Code and Naturalness
Miltiadis Allamanis, Earl T. Barr, Premkumar T. Devanbu, and Charles Sutton. 2018b · 2018
Earlier work this paper cites.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al · 2018
Earlier work this paper cites.
Selection via proxy: Efficient data selection for deep learning
Original
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. 2019 · 2019
Earlier work this paper cites.
The curious case of neural text degeneration
Original
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019 · 2019
Earlier work this paper cites.
Codesearchnet challenge: Evaluating the state of semantic code search
Original
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019 · 2019
Earlier work this paper cites.
Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension
Original
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019 · 2019
Earlier work this paper cites.
Sentence-bert: Sentence embeddings using siamese bert-networks
Original
Nils Reimers and Iryna Gurevych. 2019 · 2019
Earlier work this paper cites.
A simple framework for contrastive learning of visual representations. In International conference on machine learning . PMLR, 1597–1607
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020 · 2020
Earlier work this paper cites.
Codebert: A pre-trained model for programming and natural languages
Original
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al · 2020
Earlier work this paper cites.

Similar

Graphcodebert: Pre-training code representations with data flow
Original
Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al · 2020
Cited alongside, same era.
Exploring the limits of transfer learning with a unified text-to-text transformer
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al · 2020
Cited alongside, same era.
Codebleu: a method for automatic evaluation of code synthesis
Original
Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020 · 2020
Cited alongside, same era.
Unified pre-training for program understanding and generation.. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
WU Ahmad, S Chakraborty, B Ray, and KW Chang. 2021 · 2021
Cited alongside, same era.

Then

Fault-aware neural code rankers
Jeevana Priya Inala, Chenglong Wang, Mei Yang, Andres Codas, Mark Encarnación, Shuvendu Lahiri, Madanlal Musuvathi, and Jianfeng Gao. 2022 · 2022
Later among the works it cites.
Diverse demonstrations improve in-context compositional generalization
Original
Itay Levy, Ben Bogin, and Jonathan Berant. 2022 · 2022
Later among the works it cites.
Competition-level code generation with alphacode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al · 2022
Later among the works it cites.
Selective annotation makes language models better few-shot learners
Original
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al · 2022
Later among the works it cites.
CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
Original

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Program synthesis with large language models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al · 2021

Cited alongside, same era.

Evaluating large language models trained on code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021

Cited alongside, same era.

What Makes Good In-Context Examples for GPT- 3 3 ?

Original

Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021 · 2021

Cited alongside, same era.

Codexglue: A machine learning benchmark dataset for code understanding and generation

Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al · 2021

Cited alongside, same era.

Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation

Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021 · 2021

Cited alongside, same era.

Multi-lingual Evaluation of Code Generation Models

Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al · 2022

Cited alongside, same era.

Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code

Original

Patrick Bareiß, Beatriz Souza, Marcelo d’Amorim, and Michael Pradel. 2022 · 2022

Cited alongside, same era.

Natgen: generative pre-training by “naturalizing” source code. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 18–30

Saikat Chakraborty, Toufique Ahmed, Yangruibo Ding, Premkumar T Devanbu, and Baishakhi Ray. 2022 · 2022

Cited alongside, same era.

Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. 2022 · 2022

Later among the works it cites.

CodeGen-Test: An Automatic Code Generation Model Integrating Program Test Information

Original

Maosheng Zhong, Gen Liu, Hongwei Li, Jiangling Kuang, Jinshan Zeng, and Mingwen Wang. 2022 · 2022

Later among the works it cites.

Teaching large language models to self-debug

Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023 · 2023

Closest in time.

Impact of code language models on automated program repair

Original

Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023b · 2023

Closest in time.

Self-planning code generation with large language model

Xue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. 2023a · 2023

Closest in time.

Structured Chain-of-Thought Prompting for Code Generation

Jia Li, Yongmin Li, Ge Li, and Zhi Jin. 2023a · 2023

Closest in time.

SkCoder: A Sketch-based Approach for Automatic Code Generation. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2124–2135

Jia Li, Yongmin Li, Ge Li, Zhi Jin, Yiyang Hao, and Xing Hu. 2023b · 2023

Closest in time.

AceCoder: Utilizing Existing Code to Enhance Code Generation

Jia Li, Yunfei Zhao, Yongmin Li, Ge Li, and Zhi Jin. 2023c · 2023

Closest in time.

Dr. ICL: Demonstration-Retrieved In-context Learning

Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. 2023 · 2023

Closest in time.

Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x

Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al · 2023

Closest in time.