2023

How Effective are Large Language Models in Generating Software Specifications?

Xie, Danning, Yoo, Byungwoo, Jiang, Nan et al.

Understand

Software specifications are essential for many Software Engineering (SE) tasks such as bug detection and test generation.

Many existing approaches are proposed to extract the specifications defined in natural language form (e.g., comments) into formal machine readable form (e.g., first order logic).
However, existing approaches suffer from limited generalizability and require manual efforts.
The recent emergence of Large Language Models (LLMs), which have been successfully applied to numerous SE tasks, offers a promising avenue for automating this process.

Built on

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR , vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
Original
1907
Earlier work this paper cites.
N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, 11 2019. [Online]. Available: http://arxiv.org/abs/1908.10084
Original
1908
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” CoRR , vol. abs/1910.10683, 2019. [Online]. Available: http://arxiv.org/abs/1910.10683
Original
1910
Earlier work this paper cites.
L. Breiman and P. Spector, “Submodel selection and evaluation in regression. the x-random case,” International statistical review/revue internationale de Statistique , pp. 291–319, 1992
1992
Earlier work this paper cites.
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning , ser. Springer Series in Statistics. New York, NY, USA: Springer New York Inc., 2001
2001
Earlier work this paper cites.
C. Boyapati, S. Khurshid, and D. Marinov, “Korat: Automated testing based on java predicates,” SIGSOFT Software Engineering Notes , vol. 27, no. 4, pp. 123–133, 2002
2002
Earlier work this paper cites.
L. G. Neuberg, “Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000,” Econometric Theory , vol. 19, no. 4, pp. 675–685, 2003
2003
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” CoRR , vol. abs/2005.14165, 2020. [Online]. Available: https://arxiv.org/abs/2005.14165
Original
2005
Earlier work this paper cites.
L. Tan, D. Yuan, G. Krishna, and Y. Zhou, “/*icomment: bugs or bad comments?*/,” in Symposium on Operating Systems Principles , 2007
2007
Earlier work this paper cites.
C. Cadar, D. Dunbar, and D. Engler, “KLEE: Unassisted and automatic generation of high-coverage tests for complex systems programs,” in Proceedings of the 8th USENIX conference on Operating systems design and implementation , 2008, pp. 209–224
2008
Earlier work this paper cites.
P. R. Rosenbaum, P. Rosenbaum, and Briskman, Design of observational studies . Springer, 2010, vol. 10
2010
Earlier work this paper cites.
S. H. Tan, D. Marinov, L. Tan, and G. T. Leavens, “@tcomment: Testing javadoc comments to detect comment-code inconsistencies,” 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation , pp. 260–269, 2012
2012
Earlier work this paper cites.
R. Pandita, X. Xiao, H. Zhong, T. Xie, S. Oney, and A. Paradkar, “Inferring method specifications from natural language api descriptions,” in 2012 34th international conference on software engineering (ICSE) . IEEE, 2012, pp. 815–825
2012
Earlier work this paper cites.
E. Wong, L. Zhang, S. Wang, T. Liu, and L. Tan, “Dase: Document-assisted symbolic execution for improving automated software testing,” 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1, pp. 620–631, 2015
2015
Earlier work this paper cites.
A. Goffi, A. Gorla, M. D. Ernst, and M. Pezzè, “Automatic generation of oracles for exceptional behaviors,” in Proceedings of the 25th international symposium on software testing and analysis , 2016, pp. 213–224
2016
Earlier work this paper cites.
A. Blasi, A. Goffi, K. Kuznetsov, A. Gorla, M. D. Ernst, M. Pezzè, and S. D. Castellanos, “Translating code comments to procedure specifications,” Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2018
2018
Earlier work this paper cites.
J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR , vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/1810.04805
Original
2018
Earlier work this paper cites.
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in Annual Meeting of the Association for Computational Linguistics , 2019
2019
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Earlier work this paper cites.
M. Magnusson, M. Andersen, J. Jonasson, and A. Vehtari, “Bayesian leave-one-out cross-validation for large data,” in Proceedings of the 36th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 4244–4253. [Online]. Available: https://proceedings.mlr.press/v97/magnusson19a.html
2019
Earlier work this paper cites.
J. Zhai, Y. Shi, M. Pan, G. Zhou, Y. Liu, C. Fang, S. Ma, L. Tan, and X. Zhang, “C2s: translating natural language comments to formal program specifications,” Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020
2020
Earlier work this paper cites.
T. Lv, R. Li, Y. Yang, K. Chen, X. Liao, X. Wang, P. Hu, and L. Xing, “Rtfm! automatic assumption discovery and verification derivation from library document for api misuse detection,” Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security , 2020
2020
Earlier work this paper cites.
Y. Zhou, C. Wang, X. Yan, T. Chen, S. Panichella, and H. C. Gall, “Automatic detection and repair recommendation of directive defects in java api documentation,” IEEE Transactions on Software Engineering , vol. 46, pp. 1004–1023, 2020
2020
Earlier work this paper cites.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” ArXiv , vol. abs/2005.14165, 2020
Original
2020
Earlier work this paper cites.
R. David, L. Coniglio, M. Ceccato et al. , “Qsynth-a program synthesis based approach for binary code deobfuscation,” in BAR 2020 Workshop , 2020
2020
Earlier work this paper cites.

Similar

D. Xie, Y. Li, M. Kim, H. V. Pham, L. Tan, X. Zhang, and M. W. Godfrey, “Docter: documentation-guided fuzzing for testing deep learning api functions,” Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis , 2021
2021
Cited alongside, same era.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating large language models trained on code,” CoRR , vol. abs/2107.03374, 2021. [Online]. Available: https://arxiv.org/abs/2107.03374
Original
2021
Cited alongside, same era.
W. Yue, W. Weishi, J. Shafiq, and C. H. Steven, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 , 2021
2021
Cited alongside, same era.
W. Ahmad, S. Chakraborty, B. Ray, and K.-W. Chang, “Unified pre-training for program understanding and generation,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . Online: Association for Computational Linguistics, Jun. 2021, pp. 2655–2668. [Online]. Available: https://aclanthology.org/2021.naacl-main.211
2021
Cited alongside, same era.
P. Riley, N. Constant, M. Guo, G. Kumar, D. Uthus, and Z. Parekh, “Textsettr: Few-shot text style extraction and tunable targeted restyling,” 01 2021, pp. 3786–3800
2021
Cited alongside, same era.
O. Rubin, J. Herzig, and J. Berant, “Learning to retrieve prompts for in-context learning,” ArXiv , vol. abs/2112.08633, 2021
Original
2021
Cited alongside, same era.
J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt- 3 3 ?” arXiv preprint arXiv:2101.06804 , 2021
Original
2021
Cited alongside, same era.
R. Shin, C. H. Lin, S. Thomson, C. C. Chen, S. Roy, E. A. Platanios, A. Pauls, D. Klein, J. Eisner, and B. V. Durme, “Constrained language models yield few-shot semantic parsers,” ArXiv , vol. abs/2104.08768, 2021
Original
2021
Cited alongside, same era.
B. Wang and A. Komatsuzaki, “GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model,” https://github.com/kingoflolz/mesh-transformer-jax
2021
Cited alongside, same era.
S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman, “GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow,” Mar. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.5297715
2021
Cited alongside, same era.
A. Blasi, A. Gorla, M. D. Ernst, and M. Pezzè, “Call me maybe: Using nlp to automatically generate unit test cases respecting temporal constraints,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering , 2022, pp. 1–11
2022
Cited alongside, same era.
E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “A conversational paradigm for program synthesis,” arXiv preprint , 2022
2022
Cited alongside, same era.

Then

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288 , 2023
Original
2023
Closest in time.
R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al. , “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023
Original
2023
Closest in time.
E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, “Codegen2: Lessons for training llms on programming and natural languages,” arXiv preprint arXiv:2305.02309 , 2023
Original
2023
Closest in time.
C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in International conference on software engineering (ICSE) , 2023
2023
Closest in time.
S. Abukhalaf, M. Hamdaqa, and F. Khomh, “On codex prompt engineering for ocl generation: An empirical study,” arXiv preprint arXiv:2303.16244 , 2023
Original
2023
Closest in time.
L. Beurer-Kellner, M. Fischer, and M. Vechev, “Prompting is programming: A query language for large language models,” Proceedings of the ACM on Programming Languages , vol. 7, no. PLDI, pp. 1946–1969, 2023
2023
Closest in time.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023
Original
2023
Closest in time.
Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 423–435
2023
Closest in time.
S. Kang, J. Yoon, and S. Yoo, “Large language models are few-shot testers: Exploring llm-based general bug reproduction,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) , 2023
2023
Closest in time.
X. Xu, Z. Zhang, Z. Su, Z. Huang, S. Feng, Y. Ye, N. Jiang, D. Xie, S. Cheng, L. Tan et al. , “Leveraging generative models to recover variable names from stripped binary,” arXiv preprint arXiv:2306.02546 , 2023
Original
2023
Closest in time.
W. Sun, C. Fang, Y. You, Y. Miao, Y. Liu, Y. Li, G. Deng, S. Huang, Y. Chen, Q. Zhang et al. , “Automatic code summarization via chatgpt: How far are we?” arXiv preprint arXiv:2305.12865 , 2023
Original
2023
Closest in time.
A. M. Dakhel, V. Majdinasab, A. Nikanjam, F. Khomh, M. C. Desmarais, and Z. M. J. Jiang, “Github copilot ai pair programmer: Asset or liability?” Journal of Systems and Software , vol. 203, p. 111734, 2023
2023
Closest in time.
B. Xu, T.-D. Nguyen, T. Le-Cong, T. Hoang, J. Liu, K. Kim, C. Gong, C. Niu, C. Wang, B. Le et al. , “Are we ready to embrace generative ai for software q&a?” arXiv preprint arXiv:2307.09765 , 2023
Original
2023
Closest in time.
Y. Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth et al. , “Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion,” Advances in Neural Information Processing Systems , vol. 36, 2024
2024
Closest in time.
Y. Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2024, pp. 1592–1604
2024
Closest in time.
“Artifacts,” 2024. [Online]. Available: https://github.com/lt-asset/llm4spec
2024
Closest in time.
OpenAI, “Gpt-4,” 2024. [Online]. Available: https://platform.openai.com/docs/models/gpt-%204#gpt-4-turbo-and-gpt-4
2024
Closest in time.
——, “Gpt-3.5,” 2024. [Online]. Available: https://platform.openai.com/docs/models/gpt-%204#gpt-3-5-turbo
2024
Closest in time.
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024
Original
2024
Closest in time.
D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li et al. , “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196 , 2024
Original
2024
Closest in time.
A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei et al. , “Starcoder 2 and the stack v2: The next generation,” arXiv preprint arXiv:2402.19173 , 2024
Original
2024
Closest in time.
H. Ruan, Y. Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” arXiv preprint arXiv:2408.02232 , 2024
Original
2024
Closest in time.
D. Xie, Z. Zhang, N. Jiang, X. Xu, L. Tan, and X. Zhang, “Resym: Harnessing llms to recover variable and data structure symbols from stripped binaries,” in Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security , 2024, pp. 4554–4568
2024
Closest in time.
Z. Su, X. Xu, Z. Huang, Z. Zhang, Y. Ye, J. Huang, and X. Zhang, “Codeart: Better code models by attention regularization when symbols are lacking,” Proc. ACM Softw. Eng. , vol. 1, no. FSE, Jul. 2024. [Online]. Available: https://doi.org/10.1145/3643752
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…