LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation

S. G. Hart and L. E. Staveland, “Development of nasa-tlx (task load index): Results of empirical and theoretical research,” in Advances in psychology . Elsevier, 1988, vol. 52, pp. 139–183

1988

Earlier work this paper cites.

Y. Benjamini and Y. Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,” Journal of the Royal statistical society: series B (Methodological) , vol. 57, no. 1, pp. 289–300, 1995

1995

Earlier work this paper cites.

S. G. Hart, “Nasa-task load index (nasa-tlx); 20 years later,” in Proceedings of the human factors and ergonomics society annual meeting , vol. 50, no. 9. Sage publications Sage CA: Los Angeles, CA, 2006, pp. 904–908

2006

Earlier work this paper cites.

T. Lau, “Why programming-by-demonstration systems fail: Lessons learned for usable ai,” AI Magazine , vol. 30, no. 4, pp. 65–65, 2009

2009

Earlier work this paper cites.

A. Solar-Lezama, “The sketching approach to program synthesis,” in Programming Languages and Systems , Z. Hu, Ed. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 4–13

2009

Earlier work this paper cites.

S. Jha, S. Gulwani, S. A. Seshia, and A. Tiwari, “Oracle-guided component-based program synthesis,” in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE 2010, Cape Town, South Africa, 1-8 May 2010 , J. Kramer, J. Bishop, P. T. Devanbu, and S. Uchitel, Eds. ACM, 2010, pp. 215–224. [Online]. Available: https://doi.org/10.1145/1806799.1806833

2010

Earlier work this paper cites.

S. Gulwani, “Automating string processing in spreadsheets using input-output examples,” in PoPL’11, January 26-28, 2011, Austin, Texas, USA , January 2011. [Online]. Available: https://www.microsoft.com/en-us/research/publication/automating-string-processing-spreadsheets-using-input-output-examples/

2011

Earlier work this paper cites.

M. Mayer, G. Soares, M. Grechkin, V. Le, M. Marron, O. Polozov, R. Singh, B. Zorn, and S. Gulwani, “User interaction models for disambiguation in programming by example,” in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology , 2015, pp. 291–301

2015

Earlier work this paper cites.

S. Gulwani, O. Polozov, and R. Singh, “Program synthesis,” Found. Trends Program. Lang. , vol. 4, no. 1-2, pp. 1–119, 2017. [Online]. Available: https://doi.org/10.1561/2500000010

2017

Earlier work this paper cites.

S. Jha and S. A. Seshia, “A theory of formal synthesis via inductive learning,” Acta Informatica , vol. 54, no. 7, pp. 693–726, 2017. [Online]. Available: https://doi.org/10.1007/s00236-017-0294-5

2017

Earlier work this paper cites.

V. Le, D. Perelman, O. Polozov, M. Raza, A. Udupa, and S. Gulwani, “Interactive program synthesis,” arXiv preprint arXiv:1703.03539 , 2017

Original

2017

Earlier work this paper cites.

T. Zhang, L. Lowmanstone, X. Wang, and E. L. Glassman, “Interactive program synthesis by augmented examples,” in Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology , 2020, pp. 627–648

2020

Earlier work this paper cites.

R. Ji, J. Liang, Y. Xiong, L. Zhang, and Z. Hu, “Question selection for interactive program synthesis,” in Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation , ser. PLDI 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 1143–1158. [Online]. Available: https://doi.org/10.1145/3385412.3386025

2020

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

Original

2021

Earlier work this paper cites.

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

Original

2021

Earlier work this paper cites.

K. Rahmani, M. Raza, S. Gulwani, V. Le, D. Morris, A. Radhakrishna, G. Soares, and A. Tiwari, “Multi-modal program inference: a marriage of pre-trained language models and component-based synthesis,” Proc. ACM Program. Lang. , vol. 5, no. OOPSLA, pp. 1–29, 2021. [Online]. Available: https://doi.org/10.1145/3485535

2021

Earlier work this paper cites.

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

Original

2021

Cited alongside, same era.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean, S. Petrov, and N. Fiedel, “Palm: Scaling language modeling with pathways,” 2022. [Online]. Available: https://arxiv.org/abs/2204.02311

Original

2022

Cited alongside, same era.

E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “A conversational paradigm for program synthesis,” 2022. [Online]. Available: https://arxiv.org/abs/2203.13474

Original

2022

Cited alongside, same era.

H. Mozannar, G. Bansal, A. Fourney, and E. Horvitz, “Reading between the lines: Modeling user behavior and costs in ai-assisted programming,” arXiv preprint arXiv:2210.14306 , 2022

Original

2022

Later among the works it cites.

P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems , ser. CHI EA ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491101.3519665

2022

Later among the works it cites.

N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s code suggestions,” in Proceedings of the 19th International Conference on Mining Software Repositories , 2022, pp. 1–5

2022

Later among the works it cites.

S. Imai, “Is github copilot a substitute for human pair-programming? an empirical study,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings , 2022, pp. 319–321