An Empirical Study on Challenges for LLM Application Developers

A coefficient of agreement for nominal scales

Jacob Cohen. 1960 · 1960

Earlier work this paper cites.

The measurement of observer agreement for categorical data

J Richard Landis and Gary G Koch. 1977 · 1977

Earlier work this paper cites.

Qualitative methods in empirical studies of software engineering

Carolyn B. Seaman. 1999 · 1999

Earlier work this paper cites.

A field study of API learning obstacles

Martin P Robillard and Robert DeLine. 2011 · 2011

Earlier work this paper cites.

An empirical study of bugs in machine learning systems. In 2012 IEEE 23rd International Symposium on Software Reliability Engineering . IEEE, 271–280

Ferdian Thung, Shaowei Wang, David Lo, and Lingxiao Jiang. 2012 · 2012

Earlier work this paper cites.

Improving API usability

Brad A Myers and Jeffrey Stylos. 2016 · 2016

Earlier work this paper cites.

What are mobile developers asking about? a large scale study using stack overflow

Christoffer Rosen and Emad Shihab. 2016 · 2016

Earlier work this paper cites.

What security questions do developers ask? a large-scale study of stack overflow posts

Xin-Li Yang, David Lo, Xin Xia, Zhi-Yuan Wan, and Jian-Ling Sun. 2016 · 2016

Earlier work this paper cites.

What do concurrency developers ask about? a large-scale study using stack overflow. In Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement . 1–10

Syed Ahmed and Mehdi Bagherzadeh. 2018 · 2018

Earlier work this paper cites.

An empirical study on tensorflow program bugs. In Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis . 129–140

Yuhao Zhang, Yifan Chen, Shing-Chi Cheung, Yingfei Xiong, and Lu Zhang. 2018 · 2018

Earlier work this paper cites.

Software documentation issues unveiled. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 1199–1210

Emad Aghajani, Csaba Nagy, Olga Lucero Vega-Márquez, Mario Linares-Vásquez, Laura Moreno, Gabriele Bavota, and Michele Lanza. 2019 · 2019

Earlier work this paper cites.

Why is developing machine learning applications challenging? a study on stack overflow posts. In 2019 acm/ieee international symposium on empirical software engineering and measurement (esem) . IEEE, 1–11

Moayad Alshangiti, Hitesh Sapkota, Pradeep K Murukannaiah, Xumin Liu, and Qi Yu. 2019 · 2019

Earlier work this paper cites.

Going big: a large-scale study on what big data developers ask. In Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering . 432–442

Mehdi Bagherzadeh and Raffi Khatchadourian. 2019 · 2019

Earlier work this paper cites.

An empirical study towards characterizing deep learning development and deployment across different frameworks and platforms. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 810–822

Qianyu Guo, Sen Chen, Xiaofei Xie, Lei Ma, Qiang Hu, Hongtao Liu, Yang Liu, Jianjun Zhao, and Xiaohong Li. 2019 · 2019

Earlier work this paper cites.

A comprehensive study on deep learning bug characteristics. In Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering . 510–520

Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. 2019 · 2019

Earlier work this paper cites.

An empirical study of common challenges in developing deep learning applications. In 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE) . IEEE, 104–115

Tianyi Zhang, Cuiyun Gao, Lei Ma, Michael Lyu, and Miryung Kim. 2019 · 2019

Earlier work this paper cites.

Challenges in chatbot development: A study of stack overflow posts. In Proceedings of the 17th international conference on mining software repositories . 174–185

Ahmad Abdellatif, Diego Costa, Khaled Badran, Rabe Abdalkareem, and Emad Shihab. 2020 · 2020

Earlier work this paper cites.

A comprehensive study on challenges in deploying deep learning based software. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 750–762

Zhenpeng Chen, Yanbin Cao, Yuanqiang Liu, Haoyu Wang, Tao Xie, and Xuanzhe Liu. 2020 · 2020

Earlier work this paper cites.

Taxonomy of real faults in deep learning systems. In Proceedings of the ACM/IEEE 42nd international conference on software engineering . 1110–1121

Nargiz Humbatova, Gunel Jahangirova, Gabriele Bavota, Vincenzo Riccio, Andrea Stocco, and Paolo Tonella. 2020 · 2020

Earlier work this paper cites.

Retrieval-augmented generation for knowledge-intensive nlp tasks

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al · 2020

Earlier work this paper cites.

Understanding build issue resolution in practice: symptoms and fix patterns. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 617–628

Yiling Lou, Zhenpeng Chen, Yanbin Cao, Dan Hao, and Lu Zhang. 2020 · 2020

Earlier work this paper cites.

Challenges in developing desktop web apps: a study of stack overflow and github. In 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR) . IEEE, 271–282

Gian Luca Scoccia, Patrizio Migliarini, and Marco Autili. 2021 · 2021

Earlier work this paper cites.

A comprehensive study of deep learning compiler bugs. In Proceedings of the 29th ACM Joint meeting on european software engineering conference and symposium on the foundations of software engineering . 968–980

Qingchao Shen, Haoyang Ma, Junjie Chen, Yongqiang Tian, Shing-Chi Cheung, and Xiang Chen. 2021 · 2021

Earlier work this paper cites.

An empirical study on challenges of application development in serverless computing. In Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering . 416–428

Jinfeng Wen, Zhenpeng Chen, Yi Liu, Yiling Lou, Yun Ma, Gang Huang, Xin Jin, and Xuanzhe Liu. 2021 · 2021

Earlier work this paper cites.

Understanding performance problems in deep learning systems. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 357–369

Junming Cao, Bihuan Chen, Chao Sun, Longjie Hu, Shuaihong Wu, and Xin Peng. 2022 · 2022

Earlier work this paper cites.

A survey on in-context learning

Original

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022 · 2022

Cited alongside, same era.

Large language models are zero-shot reasoners

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022 · 2022

Cited alongside, same era.

Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 1035–1047

Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, et al · 2022

Cited alongside, same era.

Barriers in front-end web development. In 2022 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . IEEE, 1–11

David I Samudio and Thomas D LaToza. 2022 · 2022

Cited alongside, same era.

Chain-of-thought prompting elicits reasoning in large language models

No more manual tests? evaluating and improving chatgpt for unit test generation

Original

Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. 2023 · 2023

Later among the works it cites.

A survey of large language models

Original

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al · 2023

Later among the works it cites.

Automatic semantic augmentation of language model prompts (for code summarization). In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–13

Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024 · 2024

Closest in time.

De-hallucinator: Iterative grounding for llm-based code completion

Original

Aryaz Eghbali and Michael Pradel. 2024 · 2024

Closest in time.