2022

Grounded Copilot: How Programmers Interact with Code-Generating Models

Barke, Shraddha, James, Michael B., Polikarpova, Nadia

Understand

Powered by recent advances in code-generating models, AI assistants like Github Copilot promise to change the face of programming forever.

But what is this new face of programming? We present the first grounded theory analysis of how programmers interact with Copilot, based on observing 20 participants--with a range of prior experience using the assistant--as they solve diverse programming tasks across four languages.
Our main finding is that interactions with programming assistants are bimodal: in acceleration mode, the programmer knows what to do next and uses Copilot to get there faster; in exploration mode, the programmer is unsure how to proceed and uses Copilot to explore their options.
Based on our theory, we provide recommendations for improving the usability of future AI programming assistants.

Built on

Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 1901
Earlier work this paper cites.
The discovery of grounded theory: strategies for qualitative research (5. paperback print ed.)
Barney G. Glaser and Anselm L. Strauss. 1967 · 1967
Earlier work this paper cites.
Stimulus structures and mental representations in expert comprehension of computer programs
Nancy Pennington. 1987 · 1987
Earlier work this paper cites.
Basics of Qualitative Reseach: Grounded Theory Procedures and Techniques
Anselm L. Strauss and Juliet Corbin. 1990 · 1990
Earlier work this paper cites.
Introduction to qualitative research methods: A guidebook and resource, 3rd ed
Steven J. Taylor and Robert Bogdan. 1998 · 1998
Earlier work this paper cites.
QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the fifth ACM SIGPLAN international conference on Functional programming (ICFP’00) . Association for Computing Machinery, 268–279
Koen Claessen and John Hughes. 2000 · 2000
Earlier work this paper cites.
Software design—cognitive aspects
Françoise Détienne and Frank Bott. 2001 · 2001
Earlier work this paper cites.
The Impact of Background and Experience on Software Inspections
Jeff Carver. 2004 · 2004
Earlier work this paper cites.
Incorporating External Knowledge through Pre-training for Natural Language to Code Generation
Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020 · 2004
Earlier work this paper cites.
Detecting Affective Flow States of Knowledge Workers Using Physiological Sensors
Original
Matthew Lee. 2020 · 2006
Earlier work this paper cites.
Motivational Effects on Self-Regulated Learning with Different Tasks
Regina Vollmeyer and Falko Rheinberg. 2006 · 2006
Earlier work this paper cites.
The SAGE Handbook of Grounded Theory
Antony Bryant and Kathy Charmaz. 2007 · 2007
Earlier work this paper cites.
Automating string processing in spreadsheets using input-output examples
Sumit Gulwani. 2011 · 2011
Earlier work this paper cites.
Thinking, fast and slow
Daniel Kahneman. 2011 · 2011
Earlier work this paper cites.
Dual-Process Theories
Donal E Carlston. 2013 · 2013
Earlier work this paper cites.
Program sketching
Armando Solar-Lezama. 2013 · 2013
Earlier work this paper cites.
Flow and the Foundations of Positive Psychology
Mihaly Csikszentmihalyi. 2014 · 2014
Earlier work this paper cites.
Curiosity, Creativity, and Surprise as Analytic Tools: Grounded Theory Method
Michael Muller. 2014 · 2014
Earlier work this paper cites.
Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation . 419–428
Veselin Raychev, Martin Vechev, and Eran Yahav. 2014 · 2014
Earlier work this paper cites.
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale
Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip J. Guo, and Robert C. Miller. 2015 · 2015
Earlier work this paper cites.

Similar

Deepcoder: Learning to write programs
Original
Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2016 · 2016
Cited alongside, same era.
Programmers Are Users Too: Human-Centered Methods for Improving Programming Tools
Brad A. Myers, Andrew J. Ko, Thomas D. LaToza, and YoungSeok Yoon. 2016 · 2016
Cited alongside, same era.
Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering (ICSE ’16) . Association for Computing Machinery, 120–131
Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. 2016 · 2016
Cited alongside, same era.
Attention Is All You Need
Original
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017 · 2017
Cited alongside, same era.
Rousillon: Scraping Distributed Hierarchical Web Data. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18) . Association for Computing Machinery, New York, NY, USA, 963–975
Sarah E. Chasins, Maria Mueller, and Rastislav Bodik. 2018 · 2018
Cited alongside, same era.
Neural-guided deductive search for real-time program synthesis from examples
Original
Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018 · 2018
Cited alongside, same era.
Programming Not Only by Example. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18) . ACM, 1114–1124
Hila Peleg, Sharon Shoham, and Eran Yahav. 2018 · 2018
Cited alongside, same era.
TabNine: AI Assistant for Development Teams
TabNine. 2018 · 2018
Cited alongside, same era.
On the fly synthesis of edit suggestions
Anders Miltner, Sumit Gulwani, Vu Le, Alan Leung, Arjun Radhakrishna, Gustavo Soares, Ashish Tiwari, and Abhishek Udupa. 2019 · 2019
Cited alongside, same era.
Live Functional Programming with Typed Holes
Cyrus Omar, Ian Voysey, Ravi Chugh, and Matthew A. Hammer. 2019 · 2019
Cited alongside, same era.
Wrex: A Unified Programming-by-Example Interaction for Synthesizing Readable Code for Data Scientists
Ian Drosos, Titus Barik, Philip J. Guo, Robert DeLine, and Sumit Gulwani. 2020 · 2020
Cited alongside, same era.
Small-Step Live Programming by Example. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology . Association for Computing Machinery, New York, NY, USA, 614–626
Kasra Ferdowsifard, Allen Ordookhanians, Hila Peleg, Sorin Lerner, and Nadia Polikarpova. 2020 · 2020
Cited alongside, same era.

Then

Nat Friedman. 2021
2021
Later among the works it cites.
Learning to Complete Code with Sketches. In International Conference on Learning Representations
Daya Guo, Alexey Svyatkovskiy, Jian Yin, Nan Duan, Marc Brockschmidt, and Miltiadis Allamanis. 2021 · 2021
Later among the works it cites.
How statically-typed functional programmers write code
Justin Lubin and Sarah E. Chasins. 2021 · 2021
Later among the works it cites.
A rational reinterpretation of dual-process theories
Smitha Milli, Falk Lieder, and Thomas L. Griffiths. 2021 · 2021
Later among the works it cites.
reCode: A Lightweight Find-and-Replace Interaction in the IDE for Transforming Code by Example. In The 34th Annual ACM Symposium on User Interface Software and Technology . 258–269
Wode Ni, Joshua Sunshine, Vu Le, Sumit Gulwani, and Titus Barik. 2021 · 2021
Later among the works it cites.
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Original
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021 · 2021
Later among the works it cites.
Falx: Synthesis-Powered Visualization Authoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21) . Association for Computing Machinery, New York, NY, USA, Article 106, 15 pages
Chenglong Wang, Yu Feng, Rastislav Bodik, Isil Dillig, Alvin Cheung, and Amy J Ko. 2021 · 2021
Later among the works it cites.
Advent of Code
Eric Wastl. 2021 · 2021
Later among the works it cites.
Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces . Association for Computing Machinery, New York, NY, USA, 402–412
Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021 · 2021
Later among the works it cites.
In-IDE Code Generation from Natural Language: Promise and Challenges
Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2021 · 2021
Later among the works it cites.
Interpretable Program Synthesis. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21) . Association for Computing Machinery, New York, NY, USA, Article 105, 16 pages
Tianyi Zhang, Zhiyang Chen, Yuanli Zhu, Priyan Vaithilingam, Xinyu Wang, and Elena L. Glassman. 2021 · 2021
Later among the works it cites.
Efficient Training of Language Models to Fill in the Middle
Original
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022 · 2022
Closest in time.
InCoder: A Generative Model for Code Infilling and Synthesis
Original
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022 · 2022
Closest in time.
Exploring the Learnability of Program Synthesizers by Novice Programmers
Dhanya Jayagopal, Justin Lubin, and Sarah E Chasins. 2022 · 2022
Closest in time.
Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In CHI Conference on Human Factors in Computing Systems . 1–19
Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022 · 2022
Closest in time.
Competition-Level Code Generation with AlphaCode
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022 · 2022
Closest in time.
What is it like to program with artificial intelligence?
Original
Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022 · 2022
Closest in time.
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In CHI Late-Breaking Work
Priyan Vaithilingam, Tianyi Zhang, and Elena Glassman. 2022 · 2022
Closest in time.
Synthesizing Analytical SQL Queries from Computation Demonstration. In PLDI
Xiangyu Zhou, Ras Bodik, Alvin Cheung, and Chenglong Wang. 2022 · 2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…