2021

Choose Your Programming Copilot: A Comparison of the Program Synthesis Performance of GitHub Copilot and Genetic Programming

Sobania, Dominik, Briesch, Martin, Rothlauf, Franz

Understand

GitHub Copilot, an extension for the Visual Studio Code development environment powered by the large-scale language model Codex, makes automatic program synthesis available for software developers.

This model has been extensively studied in the field of deep learning, however, a comparison to genetic programming, which is also known for its performance in automatic program synthesis, has not yet been carried out.
In this paper, we evaluate GitHub Copilot on standard program synthesis benchmark problems and compare the achieved results with those from the genetic programming literature.
In addition, we discuss the performance of both approaches.

Built on

A representation for the adaptive generation of simple sequential programs. In proceedings of an International Conference on Genetic Algorithms and the Applications . 183–187
Nichael Lynn Cramer. 1985 · 1985
Earlier work this paper cites.
Genetic programming: on the programming of computers by means of natural selection . Vol. 1
John R Koza. 1992 · 1992
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber. 1997 · 1997
Earlier work this paper cites.
Genetic programming and autoconstructive evolution with the push programming language
Lee Spector and Alan Robinson. 2002 · 2002
Earlier work this paper cites.
The push3 execution stack and the evolution of control. In Proceedings of the 7th annual conference on Genetic and evolutionary computation . 1689–1696
Lee Spector, Jon Klein, and Maarten Keijzer. 2005 · 2005
Earlier work this paper cites.
Linear genetic programming
Markus F Brameier and Wolfgang Banzhaf. 2007 · 2007
Earlier work this paper cites.
Dimensions in program synthesis. In Proceedings of the 12th international ACM SIGPLAN symposium on Principles and practice of declarative programming . 13–24
Sumit Gulwani. 2010 · 2010
Earlier work this paper cites.
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 1724–1734
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014 · 2014
Earlier work this paper cites.
Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , Yoshua Bengio and Yann LeCun (Eds.)
Original
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015 · 2015
Earlier work this paper cites.
General program synthesis benchmark suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation . 1039–1046
Thomas Helmuth and Lee Spector. 2015 · 2015
Earlier work this paper cites.
Effective Approaches to Attention-based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing . 1412–1421
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015 · 2015
Earlier work this paper cites.
A grammar design pattern for arbitrary program synthesis problems in genetic programming. In European Conference on Genetic Programming . Springer, 262–277
Stefan Forstenlechner, David Fagan, Miguel Nicolau, and Michael O’Neill. 2017 · 2017
Earlier work this paper cites.

Similar

Improving generalization of evolved programs through automatic simplification. In Proceedings of the Genetic and Evolutionary Computation Conference . 937–944
Thomas Helmuth, Nicholas Freitag McPhee, Edward Pantridge, and Lee Spector. 2017 · 2017
Cited alongside, same era.
Attention is all you need. In Advances in neural information processing systems . 5998–6008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Extending program synthesis grammars for grammar-guided genetic programming. In International Conference on Parallel Problem Solving from Nature . Springer, 197–208
Stefan Forstenlechner, David Fagan, Miguel Nicolau, and Michael O’Neill. 2018 · 2018
Cited alongside, same era.
Program synthesis using uniform mutation by addition and deletion. In Proceedings of the Genetic and Evolutionary Computation Conference . 1127–1134
Thomas Helmuth, Nicholas Freitag McPhee, and Lee Spector. 2018 · 2018
Cited alongside, same era.
Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 328–339
Jeremy Howard and Sebastian Ruder. 2018 · 2018
Cited alongside, same era.
Improving language understanding by generative pre-training
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018 · 2018
Cited alongside, same era.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT . 4171–4186
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019 · 2019
Cited alongside, same era.
On domain knowledge and novelty to improve program synthesis performance with grammatical evolution. In Proceedings of the Genetic and Evolutionary Computation Conference . 1039–1046
Erik Hemberg, Jonathan Kelly, and Una-May O’Reilly. 2019 · 2019
Cited alongside, same era.
Tag-accessed memory for genetic programming. In Proceedings of the Genetic and Evolutionary Computation Conference Companion . 346–347
Alexander Lalejini and Charles Ofria. 2019 · 2019
Cited alongside, same era.
Teaching GP to program like a human software developer: using perplexity pressure to guide program synthesis approaches. In Proceedings of the Genetic and Evolutionary Computation Conference . 1065–1074
Dominik Sobania and Franz Rothlauf. 2019 · 2019
Cited alongside, same era.
Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020 · 2020
Cited alongside, same era.
PyMT5: Multi-mode Translation of Natural Language and Python Code with Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 9052–9065
Colin Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020 · 2020
Cited alongside, same era.

Then

CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings . 1536–1547
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al · 2020
Later among the works it cites.
Benchmarking parent selection for program synthesis by genetic programming. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion . 237–238
Thomas Helmuth and Amr Abdelhady. 2020 · 2020
Later among the works it cites.
SourceFinder: Finding Malware Source-Code from Publicly Available Repositories in GitHub. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020) . USENIX Association, San Sebastian, 149–163
Md Omar Faruk Rokon, Risul Islam, Ahmad Darki, Evangelos E. Papalexakis, and Michalis Faloutsos. 2020 · 2020
Later among the works it cites.
Challenges of program synthesis with grammatical evolution. In European Conference on Genetic Programming (Part of EvoStar) . Springer, 211–227
Dominik Sobania and Franz Rothlauf. 2020 · 2020
Later among the works it cites.
TensorGP-Genetic Programming Engine in TensorFlow.. In EvoApplications . Springer, 763–778
Francisco Baeta, João Correia, Tiago Martins, and Penousal Machado. 2021 · 2021
Closest in time.
Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 2633–2650
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021 · 2021
Closest in time.
Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al · 2021
Closest in time.
PSB2: the second program synthesis benchmark suite. In Proceedings of the Genetic and Evolutionary Computation Conference . 785–794
Thomas Helmuth and Peter Kelly. 2021 · 2021
Closest in time.
Problem-solving benefits of down-sampled lexicase selection
Original
Thomas Helmuth and Lee Spector. 2021 · 2021
Closest in time.
On the generalizability of programs synthesized by grammar-guided genetic programming. In European Conference on Genetic Programming (Part of EvoStar) . Springer, 130–145
Dominik Sobania. 2021 · 2021
Closest in time.
A generalizability measure for program synthesis with genetic programming. In Proceedings of the Genetic and Evolutionary Computation Conference . 822–829
Dominik Sobania and Franz Rothlauf. 2021 · 2021
Closest in time.
Recent Developments in Program Synthesis with Evolutionary Algorithms
Original
Dominik Sobania, Dirk Schweim, and Franz Rothlauf. 2021 · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…