2023

FRANC: A Lightweight Framework for High-Quality Code Generation

Siddiq, Mohammed Latif, Casey, Beatrice, Santos, Joanna C. S.

Understand

In recent years, the use of automated source code generation utilizing transformer-based generative models has expanded, and these models can generate functional code according to the requirements of the developers.

However, recent research revealed that these automatically generated source codes can contain vulnerabilities and other quality issues.
Despite researchers' and practitioners' attempts to enhance code generation models, retraining and fine-tuning large language models is time-consuming and resource-intensive.
Thus, we describe FRANC, a lightweight framework for recommending more secure and high-quality source code derived from transformer-based code generation models.

Built on

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901
1901
Earlier work this paper cites.
C. Green, “Application of theorem proving to problem solving,” in Proc. of the 1st Intl. Joint Conf. on Artificial Intelligence , ser. IJCAI’69. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1969, p. 219–239
1969
Earlier work this paper cites.
Z. Manna and R. J. Waldinger, “Toward automatic program synthesis,” Commun. ACM , vol. 14, no. 3, p. 151–165, mar 1971
1971
Earlier work this paper cites.
M. Fowler, Refactoring: Improving the Design of Existing Code . Boston, MA, USA: Addison-Wesley, 1999
1999
Earlier work this paper cites.
K. Järvelin and J. Kekäläinen, “Ir evaluation methods for retrieving highly relevant documents,” in Proc. of the 23rd Annual Intl. ACM SIGIR Conf. on Research and Development in Information Retrieval , ser. SIGIR ’00. New York, NY, USA: Association for Computing Machinery, 2000, p. 41–48
2000
Earlier work this paper cites.
——, “Cumulated gain-based evaluation of ir techniques,” ACM Trans. Inf. Syst. , vol. 20, no. 4, p. 422–446, oct 2002
2002
Earlier work this paper cites.
S. H. Kan, Metrics and models in software quality engineering . Addison-Wesley Professional, 2003
2003
Earlier work this paper cites.
M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica , vol. 22, no. 3, pp. 276–282, 2012
2012
Earlier work this paper cites.
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” 2014
2014
Earlier work this paper cites.
M. M. Najafabadi, F. Villanustre, T. M. Khoshgoftaar, N. Seliya, R. Wald, and E. Muharemagic, “Deep learning applications and challenges in big data analytics,” Journal of Big Data , vol. 2, no. 1, p. 1, Feb 2015
2015
Earlier work this paper cites.
T. Sharma, M. Fragkoulis, and D. Spinellis, “House of cards: code smells in open-source c# repositories,” in 2017 ACM/IEEE Intl. Symposium on Empirical Software Engineering and Measurement (ESEM) . IEEE, 2017, pp. 424–429
2017
Earlier work this paper cites.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017
2017
Earlier work this paper cites.
M. Ghafari, P. Gadient, and O. Nierstrasz, “Security smells in android,” in 2017 IEEE 17th Intl. working Conf. on source code analysis and manipulation (SCAM) . IEEE, 2017, pp. 121–130
2017
Earlier work this paper cites.
S. Gulwani, O. Polozov, R. Singh et al. , “Program synthesis,” Foundations and Trends® in Programming Languages , vol. 4, no. 1-2, pp. 1–119, 2017
2017
Earlier work this paper cites.
P. Yin and G. Neubig, “A syntactic neural model for general-purpose code generation,” in Proc. of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 440–450
2017
Earlier work this paper cites.
F. Palomba, G. Bavota, M. D. Penta, F. Fasano, R. Oliveto, and A. D. Lucia, “On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigation,” Empirical Software Engineering , vol. 23, no. 3, pp. 1188–1221, 2018
2018
Earlier work this paper cites.
A. Gupta, B. Suri, V. Kumar, S. Misra, T. Blažauskas, and R. Damaševičius, “Software code smell prediction model using shannon, rényi and tsallis entropies,” Entropy , vol. 20, no. 5, p. 372, 2018
2018
Earlier work this paper cites.
D. A. Tomassi, “Bugs in the wild: examining the effectiveness of static analyzers at finding real-world bugs,” in Proc. of the 2018 26th ACM Joint Meeting on European Software Engineering Conf. and Symposium on the Foundations of Software Engineering , 2018, pp. 980–982
2018
Earlier work this paper cites.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) . Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186
2019
Earlier work this paper cites.
A. Rahman, C. Parnin, and L. Williams, “The Seven Sins: Security Smells in Infrastructure as Code Scripts,” in 2019 IEEE/ACM 41st Intl. Conf. on Software Engineering (ICSE) . Montreal, QC, Canada: IEEE, May 2019, pp. 164–175
2019
Earlier work this paper cites.
M. R. Rahman, A. Rahman, and L. Williams, “Share, but be aware: Security smells in python gists,” in 2019 IEEE Intl. Conf. on Software Maintenance and Evolution (ICSME) , 2019, pp. 536–540
2019
Earlier work this paper cites.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
Earlier work this paper cites.
T. H. M. Le, H. Chen, and M. A. Babar, “Deep learning for source code modeling and generation: Models, applications, and challenges,” ACM Comput. Surv. , vol. 53, no. 3, jun 2020
2020
Earlier work this paper cites.
A. Sherstinsky, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network,” Physica D: Nonlinear Phenomena , vol. 404, p. 132306, mar 2020
2020
Earlier work this paper cites.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research , vol. 21, no. 140, pp. 1–67, 2020
2020
Earlier work this paper cites.

Similar

Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “CodeBERT: A pre-trained model for programming and natural languages,” in Findings of the Association for Computational Linguistics: EMNLP 2020 . Online: Association for Computational Linguistics, Nov. 2020, pp. 1536–1547
2020
Cited alongside, same era.
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto et al. , “Evaluating large language models trained on code,” 2021
2021
Cited alongside, same era.
K. W. Church, Z. Chen, and Y. Ma, “Emerging trends: A gentle introduction to fine-tuning,” Natural Language Engineering , vol. 27, no. 6, p. 763–778, 2021
2021
Cited alongside, same era.
S. Kim, J. Zhao, Y. Tian, and S. Chandra, “Code prediction by feeding trees to transformers,” in 2021 IEEE/ACM 43rd Intl. Conf. on Software Engineering (ICSE) . IEEE, 2021, pp. 150–162
2021
Cited alongside, same era.
A. Svyatkovskiy, S. Lee, A. Hadjitofi, M. Riechert, J. V. Franco, and M. Allamanis, “Fast and memory-efficient neural code completion,” in 2021 IEEE/ACM 18th Intl. Conf. on Mining Software Repositories (MSR) . IEEE, 2021, pp. 329–340
2021
Cited alongside, same era.
Y. Wang, W. Wang, S. Joty, and S. C. Hoi, “CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” in Proc. of the 2021 Conf. on Empirical Methods in Natural Language Processing . Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 8696–8708
2021
Cited alongside, same era.
A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig, “Language models of code are few-shot commonsense learners,” 2022. [Online]. Available: https://arxiv.org/abs/2210.07128
Original
2022
Cited alongside, same era.
B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “Codet: Code generation with generated tests,” 2022. [Online]. Available: https://arxiv.org/abs/2207.10397
Original
2022
Cited alongside, same era.
H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi, “Coderl: Mastering code generation through pretrained models and deep reinforcement learning,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 21 314–21 328
2022
Cited alongside, same era.
M. L. Siddiq, S. H. Majumder, M. R. Mim, S. Jajodia, and J. C. Santos, “An empirical study of code smells in transformer-based code generation techniques,” in 2022 IEEE 22nd Intl. Working Conf. on Source Code Analysis and Manipulation (SCAM) , 2022
2022
Cited alongside, same era.
H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? assessing the security of github copilot’s code contributions,” in 2022 IEEE Symposium on Security and Privacy (SP) , 2022, pp. 754–768
2022
Cited alongside, same era.
G. Sandoval, H. Pearce, T. Nys, R. Karri, B. Dolan-Gavitt, and S. Garg, “Security implications of large language model code assistants: A user study,” arXiv preprint arXiv:2208.09727 , 2022
Original
2022
Cited alongside, same era.

Then

J. P. Inala, C. Wang, M. Yang, A. Codas, M. Encarnación, S. Lahiri, M. Musuvathi, and J. Gao, “Fault-aware neural code rankers,” in Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 13 419–13 432
2022
Later among the works it cites.
F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proc. of the 6th ACM SIGPLAN Intl. Symposium on Machine Programming , ser. MAPS 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 1–10
2022
Later among the works it cites.
T. M. C. (MITRE), “Common weakness enumeration,” 2022, [Online; accessed 18. Aug. 2022]. [Online]. Available: https://cwe.mitre.org/
2022
Later among the works it cites.
L. von Werra, “Codeparrot,” 2022. [Online]. Available: https://github.com/huggingface/transformers/tree/main/examples/research_projects/codeparrot
2022
Later among the works it cites.
D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W. tau Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” CoRR , vol. abs/2204.05999, 2022
Original
2022
Later among the works it cites.
B. Developers, “Bandit,” 2022, [Online; accessed 28. June. 2022]. [Online]. Available: https://bandit.readthedocs.io/
2022
Later among the works it cites.
D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, W. Yongji, and J.-G. Lou, “Large language models meet nl2code: A survey,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 2023, pp. 7443–7464
2023
Closest in time.
S. Gunasekar, Y. Zhang, J. Aneja, C. C. T. Mendes, A. Del Giorno, S. Gopi, M. Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi et al. , “Textbooks are all you need,” arXiv preprint arXiv:2306.11644 , 2023
Original
2023
Closest in time.
OpenAI, “Openai pricing,” Accessed May 5, 2023, 2023. [Online]. Available: https://openai.com/pricing
2023
Closest in time.
J. He and M. Vechev, “Large language models for code: Security hardening and adversarial testing,” 2023
2023
Closest in time.
A. Chen, J. Scheurer, T. Korbak, J. A. Campos, J. S. Chan, S. R. Bowman, K. Cho, and E. Perez, “Improving code generation by training with natural language feedback,” 2023
2023
Closest in time.
H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, T. Xie, and Q. Wang, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” 2023
2023
Closest in time.
B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang, “Multi-lingual evaluation of code generation models,” in The Eleventh Intl. Conf. on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=Bo7eeXm6An8
2023
Closest in time.
S. Barke, M. B. James, and N. Polikarpova, “Grounded copilot: How programmers interact with code-generating models,” Proceedings of the ACM on Programming Languages , vol. 7, no. OOPSLA1, pp. 85–111, 2023
2023
Closest in time.
H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt, “Examining zero-shot vulnerability repair with large language models,” in 2023 IEEE Symposium on Security and Privacy (SP) . Los Alamitos, CA, USA: IEEE Computer Society, may 2023, pp. 1–18
2023
Closest in time.
Y. Dong, J. Ding, X. Jiang, Z. Li, G. Li, and Z. Jin, “Codescore: Evaluating code generation by learning code execution,” 2023
2023
Closest in time.
S. Fakhoury, S. Chakraborty, M. Musuvathi, and S. K. Lahiri, “Towards generating functionally correct code edits from natural language issue descriptions,” 2023
2023
Closest in time.
J. Li, Y. Li, G. Li, Z. Jin, Y. Hao, and X. Hu, “Skcoder: A sketch-based approach for automatic code generation,” 2023
2023
Closest in time.
OpenAI, “Chat completions,” Accessed Mar 25, 2023, 2023. [Online]. Available: https://platform.openai.com/docs/guides/chat
2023
Closest in time.
SpotBugs Team, “Spotbugs: Find bugs in java programs,” Accessed Mar 25, 2023, 2023. [Online]. Available: https://spotbugs.github.io
2023
Closest in time.
Y. Ding, M. J. Min, G. Kaiser, and B. Ray, “Cycle: Learning to self-refine the code generation,” Proc. ACM Program. Lang. , vol. 8, no. OOPSLA1, apr 2024
2024
Closest in time.
D. Wu, W. U. Ahmad, D. Zhang, M. K. Ramanathan, and X. Ma, “Repoformer: Selective retrieval for repository-level code completion,” arXiv preprint arXiv:2403.10059 , 2024
Original
2024
Closest in time.
M. L. Siddiq, J. C. S. Santos, R. H. Tanvir, N. Ulfat, F. A. Rifat, and V. C. Lopes, “Using large language models to generate junit tests: An empirical study,” in 28th International Conference on Evaluation and Assessment in Software Engineering (EASE 2024) , 2024
2024
Closest in time.
C. Negri-Ribalta, R. Geraud-Stewart, A. Sergeeva, and G. Lenzini, “A systematic literature review on the impact of ai models on the security of code generation,” Frontiers in Big Data , vol. 7, p. 1386720, 2024
2024
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…