On the Challenges of Fuzzing Techniques via Large Language Models

R. Isermann, J. Schaffnit, and S. Sinsel, “Hardware-in-the-loop simulation for the design and testing of engine-control systems,” IFAC Proceedings Volumes , vol. 31, no. 4, pp. 1–10, 1998, 5th IFAC Workshop on Algorithms & Architecture for Real Time Control (AARTC’98), Cancun, Mexico, 15-17 April 1998. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1474667017421252

Original

1998

Earlier work this paper cites.

M. Sokolova, N. Japkowicz, and S. Szpakowicz, “Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation,” in Australasian joint conference on artificial intelligence . Springer, 2006, pp. 1015–1021

2006

Earlier work this paper cites.

M. E. Khan and F. Khan, “A comparative study of white box, black box and grey box testing techniques,” International Journal of Advanced Computer Science and Applications , vol. 3, no. 6, 2012

2012

Earlier work this paper cites.

B. Romera-Paredes and P. Torr, “An embarrassingly simple approach to zero-shot learning,” in Proceedings of the 32nd International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, F. Bach and D. Blei, Eds., vol. 37. Lille, France: PMLR, 07–09 Jul 2015, pp. 2152–2161. [Online]. Available: https://proceedings.mlr.press/v37/romera-paredes15.html

2015

Earlier work this paper cites.

M. Böhme, V.-T. Pham, and A. Roychoudhury, “Coverage-based greybox fuzzing as markov chain,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security , 2016, pp. 1032–1043

2016

Earlier work this paper cites.

M. Böhme, V.-T. Pham, M.-D. Nguyen, and A. Roychoudhury, “Directed greybox fuzzing,” in Proceedings of the 2017 ACM SIGSAC conference on computer and communications security , 2017, pp. 2329–2344

2017

Earlier work this paper cites.

C. Chen, B. Cui, J. Ma, R. Wu, J. Guo, and W. Liu, “A systematic review of fuzzing techniques,” Computers & Security , vol. 75, pp. 118–137, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0167404818300658

2018

Earlier work this paper cites.

J. Li, B. Zhao, and C. Zhang, “Fuzzing: a survey,” Cybersecurity , vol. 1, pp. 1–13, 2018

2018

Earlier work this paper cites.

H. Liang, X. Pei, X. Jia, W. Shen, and J. Zhang, “Fuzzing: State of the art,” IEEE Transactions on Reliability , vol. 67, no. 3, pp. 1199–1218, 2018

2018

Earlier work this paper cites.

G. Klees, A. Ruef, B. Cooper, S. Wei, and M. Hicks, “Evaluating fuzz testing,” 2018

2018

Earlier work this paper cites.

C. Lemieux and K. Sen, “Fairfuzz: A targeted mutation strategy for increasing greybox fuzz testing coverage,” in 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2018, pp. 475–485

2018

Earlier work this paper cites.

J. Wang, B. Chen, L. Wei, and Y. Liu, “Superion: Grammar-aware greybox fuzzing,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) , 2019, pp. 724–735

2019

Earlier work this paper cites.

H. M. Pandey and D. Windridge, “A comprehensive classification of deep learning libraries,” in Third International Congress on Information and Communication Technology: ICICT 2018, London . Springer, 2019, pp. 427–435

2019

Earlier work this paper cites.

F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” 2019

2019

Earlier work this paper cites.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” 2020

2020

Earlier work this paper cites.

Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” ACM computing surveys (csur) , vol. 53, no. 3, pp. 1–34, 2020

2020

Earlier work this paper cites.

Z. Wang, M. Yan, J. Chen, S. Liu, and D. Zhang, “Deep learning library testing via effective model generation,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 788–799

2020

Earlier work this paper cites.

V.-T. Pham, M. Böhme, and A. Roychoudhury, “Aflnet: a greybox fuzzer for network protocols,” in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) . IEEE, 2020, pp. 460–465

2020

Earlier work this paper cites.

Y. Lin, Y. Guan, A. Asudeh, and H. Jagadish, “Identifying insufficient data coverage in databases with multiple relations,” Proceedings of the VLDB Endowment , vol. 13, no. 11, 2020

2020

Earlier work this paper cites.

V. J. Manès, H. Han, C. Han, S. K. Cha, M. Egele, E. J. Schwartz, and M. Woo, “The art, science, and engineering of fuzzing: A survey,” IEEE Transactions on Software Engineering , vol. 47, no. 11, pp. 2312–2331, 2021

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan et al. , “Evaluating large language models trained on code,” 2021. [Online]. Available: https://arxiv.org/abs/2107.03374

Original

2021

Earlier work this paper cites.

M. Chen, J. Tworek, H. Jun, Q. Yuan et al. , “Evaluating large language models trained on code,” 2021

2021

Earlier work this paper cites.

R. Natella and V.-T. Pham, “Profuzzbench: a benchmark for stateful protocol fuzzing,” in Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2021. New York, NY, USA: Association for Computing Machinery, 2021, p. 662–665. [Online]. Available: https://doi.org/10.1145/3460319.3469077

2021

Earlier work this paper cites.

J. Metzman, L. Szekeres, L. Simon, R. Sprabery, and A. Arya, “Fuzzbench: an open fuzzer benchmarking platform and service,” in Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering , 2021, pp. 1393–1403

2021

Earlier work this paper cites.

K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi et al. , “Large language models encode clinical knowledge,” 2022. [Online]. Available: https://arxiv.org/abs/2212.13138

Original

2022

Earlier work this paper cites.

R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, “Biogpt: generative pre-trained transformer for biomedical text generation and mining,” Briefings in Bioinformatics , vol. 23, no. 6, Sep. 2022. [Online]. Available: http://dx.doi.org/10.1093/bib/bbac409

2022

Earlier work this paper cites.

R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic, “Galactica: A large language model for science,” 2022. [Online]. Available: https://arxiv.org/abs/2211.09085

Original

2022

Earlier work this paper cites.

A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra, “Solving quantitative reasoning problems with language models,” 2022. [Online]. Available: https://arxiv.org/abs/2206.14858

Original

2022

Cited alongside, same era.

X. Zhu, S. Wen, S. Camtepe, and Y. Xiang, “Fuzzing: A survey for roadmap,” ACM Comput. Surv. , vol. 54, no. 11s, Sep. 2022. [Online]. Available: https://doi.org/10.1145/3512345

2022

Cited alongside, same era.

R. Gopinath, P. Görz, and A. Groce, “Mutation analysis: Answering the fuzzing challenge,” 2022. [Online]. Available: https://arxiv.org/abs/2201.11303

Original

2022

Cited alongside, same era.

A. Wei, Y. Deng, C. Yang, and L. Zhang, “Free lunch for testing: Fuzzing deep-learning libraries from open source,” 2022

2022

Cited alongside, same era.

J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, “Challenges and applications of large language models,” 2023

2023

Later among the works it cites.

Y. Li, “Business analysis — ai computational cost,” Mar 2023. [Online]. Available: https://medium.com/geekculture/business-analysis-ai-computational-cost-67a136957c95

2023

Later among the works it cites.

J. Hu, Q. Zhang, and H. Yin, “Augmenting greybox fuzzing with generative ai,” 2023. [Online]. Available: https://arxiv.org/abs/2306.06782

Original

2023

Later among the works it cites.

OpenAI, J. Achiam, S. Adler et al. , “Gpt-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774

Original

2024

Closest in time.

S. Shetye, “An evaluation of khanmigo, a generative ai tool, as a computer-assisted language learning app,” Studies in Applied Linguistics and TESOL , vol. 24, 07 2024