2024

Harnessing the Power of LLMs: Automating Unit Test Generation for High-Performance Computing

Karanjai, Rabimba, Hussain, Aftab, Rabin, Md Rafiqul Islam et al.

Understand

Unit testing is crucial in software engineering for ensuring quality.

However, it's not widely used in parallel and high-performance computing software, particularly scientific applications, due to their smaller, diverse user base and complex logic.
These factors make unit testing challenging and expensive, as it requires specialized knowledge and existing automated tools are often ineffective.
To address this, we propose an automated method for generating unit tests for such software, considering their unique features like complex logic and parallel processing.

Built on

An empirical study of the reliability of unix utilities
Barton P Miller, Lars Fredriksen, and Bryan So · 1990
Earlier work this paper cites.
The SR Programming Language: Concurrency in Practice
G.R. Andrews and R.A. Olsson · 1993
Earlier work this paper cites.
Parallel programming with mpi, 1997
Peter S Pacheco · 1997
Earlier work this paper cites.
Using MPI: portable parallel programming with the message-passing interface , volume 1
William Gropp, Ewing Lusk, and Anthony Skjellum · 1999
Earlier work this paper cites.
Winnowing: local algorithms for document fingerprinting
Saul Schleimer, Daniel S Wilkerson, and Alex Aiken · 2003
Earlier work this paper cites.
Integrating mpi-based numerical software into an advanced parallel computing environment
Pasqua D’Ambra, Marco Danelutto, Daniela di Serafino, and Marco Lapegna · 2003
Earlier work this paper cites.
Evaluating advantages of test driven development: A controlled experiment with professionals
Gerardo Canfora, Aniello Cimitile, Felix Garcia, Mario Piattini, and Corrado Aaron Visaggio · 2006
Earlier work this paper cites.
An Introduction to Parellel Programming
TF Wittwer · 2006
Earlier work this paper cites.
Tool-assisted unit test generation and selection based on operational abstractions
Tao Xie and David Notkin · 2006
Earlier work this paper cites.
Randoop: feedback-directed random testing for java
Carlos Pacheco and Michael D Ernst · 2007
Earlier work this paper cites.
Diffgen: Automated regression unit-test generation
Kunal Taneja and Tao Xie · 2008
Earlier work this paper cites.
Understanding the high-performance-computing community: A software engineer’s perspective
Victor R. Basili, Jeffrey C. Carver, Daniela Cruzes, Lorin M. Hochstein, Jeffrey K. Hollingsworth, Forrest Shull, and Marvin V. Zelkowitz · 2008
Earlier work this paper cites.
On the effectiveness of unit test automation at microsoft
Laurie Williams, Gunnar Kudrjavets, and Nachiappan Nagappan · 2009
Earlier work this paper cites.
Optimizing the parallel scheme of the poisson solver for the reduced kinetic code teresa
Thomas Cartier-Michaud, Philippe Ghendrih, Virginie Grandgirard, and Guillaume Latu · 2013
Earlier work this paper cites.
A survey on unit testing practices and problems
Ermira Daka and Gordon Fraser · 2014
Earlier work this paper cites.
Transferring an automated test generation tool to practice: From pex to fakes and code digger
Nikolai Tillmann, Jonathan De Halleux, and Tao Xie · 2014
Earlier work this paper cites.
Sparse Matrix Multiplication: The Distributed Block-Compressed Sparse Row Library
Urban Borstnik, Joost VandeVondele, Valery Weber, and Juerg Hutter · 2014
Earlier work this paper cites.
Towards test driven development for computational science with pfunit
Michael Rilee and Thomas Clune · 2014
Earlier work this paper cites.
Towards automatic and flexible unit test generation for legacy hpc code
Christian Hovy and Julian Kunkel · 2016
Earlier work this paper cites.
On the relation between unit testing and code quality
Lucas Gren and Vard Antinyan · 2017
Earlier work this paper cites.
An industrial evaluation of unit test generation: Finding real faults in a financial application
M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds · 2017
Earlier work this paper cites.
Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space
Liwei Wang, Alexander Schwing, and Svetlana Lazebnik · 2017
Earlier work this paper cites.

Similar

On the effectiveness of unit tests in test-driven development
Ayse Tosun, Muzamil Ahmed, Burak Turhan, and Natalia Juristo · 2018
Cited alongside, same era.
An empirical investigation on the readability of manual and generated test cases
Giovanni Grano, Simone Scalabrino, Harald C Gall, and Rocco Oliveto · 2018
Cited alongside, same era.
Test-driven development in hpc science: A case study
Aziz Nanthaamornphong and Jeffrey C. Carver · 2018
Cited alongside, same era.
Coverage guided, property based testing
Leonidas Lampropoulos, Michael Hicks, and Benjamin C. Pierce · 2019
Cited alongside, same era.
Fast, diverse and accurate image captioning guided by part-of-speech
Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander G Schwing, and David Forsyth · 2019
Cited alongside, same era.
IntelliCode compose: Code generation using Transformer
Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan · 2020
Cited alongside, same era.
Unit test case generation with transformers and focal context
Original
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan · 2020
Cited alongside, same era.
GitHub Copilot: Your AI pair programmer
GitHub and OpenAI · 2021
Cited alongside, same era.
Studying the usage of text-to-text transfer transformer to support code-related tasks
Antonio Mastropaolo, Simone Scalabrino, Nathan Cooper, David Nader Palacio, Denys Poshyvanyk, Rocco Oliveto, and Gabriele Bavota · 2021
Cited alongside, same era.
(r)se challenges in hpc, 2021
Jonas Thies, Melven Röhrig-Zöllner, and Achim Basermann · 2021
Cited alongside, same era.
URL https://www.openmp.org/spec-html/5.1/openmpse30.html
Nesting of regions, 2021 · 2021
Cited alongside, same era.
Asleep at the keyboard? assessing the security of github copilot’s code contributions, 2021
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri · 2021
Cited alongside, same era.

Then

Starcoder: may the source be with you!, 2023
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries · 2023
Later among the works it cites.
The potential of llms for coding with low-resource and domain-specific programming languages
Original
Artur Tarassow · 2023
Later among the works it cites.
Microsoft’s github copilot now has over a million users, 2023
Inside · 2023
Later among the works it cites.
An empirical evaluation of using large language models for automated unit test generation
Original
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip · 2023
Later among the works it cites.
Evaluation of openai codex for hpc parallel programming models kernel generation
William Godoy, Pedro Valero-Lara, Keita Teranishi, Prasanna Balaprakash, and Jeffrey Vetter · 2023
Later among the works it cites.
Self-collaboration code generation via chatgpt
Original
Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li · 2023
Later among the works it cites.
Exploring the effectiveness of large language models in generating unit tests
Original
Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes · 2023
Later among the works it cites.
Codet5+: Open code large language models for code understanding and generation
Original
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi · 2023
Later among the works it cites.
Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang · 2023
Later among the works it cites.
URL https://github.com/kthohr/optim
Github - kthohr/optim: Optimlib: a lightweight c++ library of numerical optimization methods for nonlinear functions, 2023 · 2023
Later among the works it cites.
URL https://github.com/mratsim/Arraymancer
Github - mratsim/arraymancer: A fast, ergonomic and portable tensor library in nim with a deep learning focus for cpu, gpu and embedded devices via openmp, cuda and opencl backends, 2023 · 2023
Later among the works it cites.
URL https://github.com/OpenNMT/CTranslate2
Github - opennmt/ctranslate2: Fast inference engine for transformer models, 2023 · 2023
Later among the works it cites.
URL https://github.com/faasm/faasm
Github - faasm/faasm: High-performance stateful serverless runtime based on webassembly, 2023 · 2023
Later among the works it cites.
URL https://github.com/ddemidov/amgcl
Github - ddemidov/amgcl: C++ library for solving large sparse linear systems with algebraic multigrid method, 2023 · 2023
Later among the works it cites.
URL https://github.com/kthohr/stats
Github - kthohr/stats: A c++ header-only library of statistical distribution functions., 2023 · 2023
Later among the works it cites.
URL https://github.com/OpenCppCoverage/OpenCppCoverage
Opencppcoverage/opencppcoverage: Opencppcoverage is an open source code coverage tool for c++ under windows., 2023 · 2023
Later among the works it cites.
URL https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
Gcov (using the gnu compiler collection (gcc)), 2023 · 2023
Later among the works it cites.
URL https://github.com/TApplencourt/OvO
Tapplencourt/ovo: Openmp vs offload, 2023 · 2023
Later among the works it cites.
Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models
Caroline Lemieux, Jeevana Priya Inala, Shuvendu K Lahiri, and Siddhartha Sen · 2023
Later among the works it cites.
Large language models and simple, stupid bugs
K. Jesse, T. Ahmed, P. T. Devanbu, and E. Morgan · 2023
Later among the works it cites.
A study on robustness and reliability of large language model code generation, 2023
Li Zhong and Zilong Wang · 2023
Later among the works it cites.
No more manual tests? evaluating and improving chatgpt for unit test generation, 2023
Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…