2021

An Empirical Study of Automated Unit Test Generation for Python

Lukasczyk, Stephan, Kroiß, Florian, Fraser, Gordon

Understand

Various mature automated test generation tools exist for statically typed programming languages such as Java.

Automatically generating unit tests for dynamically typed programming languages such as Python, however, is substantially more difficult due to the dynamic nature of these languages as well as the lack of type information.
Our Pynguin framework provides automated unit test generation for Python.
In this paper, we extend our previous work on Pynguin to support more aspects of the Python language, and by studying a larger variety of well-established state of the art test-generation algorithms, namely DynaMOSA, MIO, and MOSA.

Built on

Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics 18(1):50–60, DOI 10.1214/aoms/1177730491
1947
Earlier work this paper cites.
Levenshtein VI (1966) Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady, vol 10, pp 707–710
1966
Earlier work this paper cites.
McCabe TJ (1976) A complexity measure. IEEE Transactions on Software Engineering 2(4):308–320, DOI 10.1109/TSE.1976.233837
1976
Earlier work this paper cites.
Acree AT, Budd TA, DeMillo RA, Lipton RJ, Sayward FG (1978) Mutation analysis. Tech. Rep. GIT-ICS-79/08, Georgia Institute of Technology, Atlanta, Georgia, USA
1978
Earlier work this paper cites.
DeMillo RA, Lipton RJ, Sayward FG (1978) Hints on test data selection: Help for the practicing programmer. Computer 11(4):34–41, DOI 10.1109/C-M.1978.218136
1978
Earlier work this paper cites.
Ferrante J, Ottenstein KJ, Warren JD (1987) The program dependence graph and its use in optimization. ACM Transactions on Programming Languages and Systems 9(3):319–349, DOI 10.1145/24039.24041
1987
Earlier work this paper cites.
Vargha A, Delaney HD (2000) A critique and improvement of the cl common language effect size statistics of mcgraw and wong. Journal of Educational and Behavioral Statistics 25(2):101–132
2000
Earlier work this paper cites.
Csallner C, Smaragdakis Y (2004) Jcrasher: an automatic robustness tester for java. Software: Practice and Experience 34(11):1025–1050, DOI 10.1002/spe.602
2004
Earlier work this paper cites.
Tonella P (2004) Evolutionary testing of classes. In: International Symposium on Software Testing and Analysis (ISSTA), ACM, pp 119–128, DOI 10.1145/1007512.1007528
2004
Earlier work this paper cites.
Wappler S, Lammermann F (2005) Using evolutionary algorithms for the unit testing of object-oriented software. In: Annual Conference on Genetic and Evolutionary Computation (GECCO), pp 1053–1060, DOI 10.1145/1068009.1068187
2005
Earlier work this paper cites.
Xie T (2006) Augmenting automatically generated unit-test suites with regression oracle checking. In: European Conference on Object-Oriented Programming (ECOOP), Springer, Lecture Notes in Computer Science, vol 4067, pp 380–403, DOI 10.1007/11785477\_23
2006
Earlier work this paper cites.
Pacheco C, Lahiri SK, Ernst MD, Ball T (2007) Feedback-directed random test generation. In: International Conference on Software Engineering (ICSE), IEEE Computer Society, pp 75–84, DOI 10.1109/ICSE.2007.37
2007
Earlier work this paper cites.
Andrews JH, Menzies T, Li FCH (2011) Genetic algorithms for randomized unit testing. IEEE Transactions on Software Engineering 37(1):80–94, DOI 10.1109/TSE.2010.46
2010
Earlier work this paper cites.
Baresi L, Miraz M (2010) Testful: Automatic unit-test generation for java classes. In: International Conference on Software Engineering (ICSE), ACM, vol 2, pp 281–284, DOI 10.1145/1810295.1810353
2010
Earlier work this paper cites.
Jia Y, Harman M (2011) An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering 37(5):649–678, DOI 10.1109/TSE.2010.62
2010
Earlier work this paper cites.
Artzi S, Dolby J, Jensen SH, Møller A, Tip F (2011) A framework for automated testing of JavaScript web applications. In: International Conference on Software Engineering (ICSE), ACM, pp 571–580, DOI 10.1145/1985793.1985871
2011
Earlier work this paper cites.
Fraser G, Zeller A (2012) Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering 38(2):278–292, DOI 10.1109/TSE.2011.93
2011
Earlier work this paper cites.
Mairhofer S, Feldt R, Torkar R (2011) Search-based software testing and test data generation for a dynamic programming language. In: Annual Conference on Genetic and Evolutionary Computation (GECCO), ACM, pp 1859–1866, DOI 10.1145/2001576.2001826
2011
Earlier work this paper cites.
Fraser G, Arcuri A (2012) The seed is strong: Seeding strategies in search-based software testing. In: International Conference on Software Testing, Verification and Validation (ICST), IEEE Computer Society, pp 121–130, DOI 10.1109/ICST.2012.92
2012
Earlier work this paper cites.
Fraser G, Arcuri A (2013) Whole test suite generation. IEEE Transactions on Software Engineering 39(2):276–291, DOI 10.1109/TSE.2012.14
2012
Earlier work this paper cites.
Kleinschmager S, Hanenberg S, Robbes R, Éric Tanter, Stefik A (2012) Do static type systems improve the maintainability of software systems? an empirical study. In: International Conference on Program Comprehension (ICPC), IEEE Computer Society, pp 153–162, DOI 10.1109/ICPC.2012.6240483
2012
Earlier work this paper cites.

Similar

Mayer C, Hanenberg S, Robbes R, Éric Tanter, Stefik A (2012) An empirical study of the influence of static type systems on the usability of undocumented software. In: Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), ACM, pp 683–702, DOI 10.1145/2384616.2384666
2012
Cited alongside, same era.
Arcuri A (2013) It really does matter how you normalize the branch distance in search-based software testing. Journal of Software Testing, Verification and Reliability 23(2):119–147, DOI 10.1002/stvr.457
2013
Cited alongside, same era.
Arcuri A, Fraser G (2013) Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Software Engineering 18(3):594–623, DOI 10.1107/s10664-013-9249-9
2013
Cited alongside, same era.
Meyerovich LA, Rabkin AS (2013) Empirical analysis of programming language adoption. In: Annual ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), ACM, pp 1–18, DOI 10.1145/2509136.2509515
2013
Cited alongside, same era.
Derezinska A, Hałas K (2014) Experimental evaluation of mutation testing approaches to python programs. In: International Conference on Software Testing, Verification and Validation Workshops (ICST-Workshops), IEEE Computer Society, pp 156–164, DOI 10.1109/ICSTW.2014.24
2014
Cited alongside, same era.
Just R, Jalali D, Inozemtseva L, Ernst MD, Holmes R, Fraser G (2014) Are mutants a valid substitute for real faults in software testing? In: International Symposium on Foundations of Software Engineering (FSE), ACM, pp 654–665, DOI 10.1145/2635868.2535929
2014
Cited alongside, same era.
Li G, Andreasen E, Ghosh I (2014) SymJS: Automatic symbolic testing of JavaScript web applications. In: International Symposium on Foundations of Software Engineering (FSE), ACM, pp 449–459, DOI 10.1145/2635858.2635913
2014
Cited alongside, same era.
von Lücken C, Barán B, Brizuela CA (2014) A survey on multi-objective evolutionary algorithms for many-objective problems. Computational Optimization and Applications 58(3):707–756, DOI 10.1007/s10589-014-9644-1
2014
Cited alongside, same era.
Sakti A, Pesant G, Guéhéneuc Y (2015) Instance generator and problem representation to improve object oriented code coverage. IEEE Transactions on Software Engineering 41(3):294–313, DOI 10.1109/TSE.2014.2363479
2014
Cited alongside, same era.
Gong L, Pradel M, Sridharan M, Sen K (2015) DLint: dynamically checking bad coding practices in JavaScript. In: International Symposium on Software Testing and Analysis (ISSTA), ACM, pp 94–105, DOI 10.1145/2771783.2771809
2015
Cited alongside, same era.
Groce A, Pinto J (2015) A little language for testing. In: NASA International Symposium on Formal Methods (NFM), Springer, Lecture Notes in Computer Science, vol 9058, pp 204–218, DOI 10.1007/978-3-319-17524-9\_15
2015
Cited alongside, same era.
Ma L, Artho C, Zhang C, Sato H, Gmeiner J, Ramler R (2015) GRT: program-analysis-guided random testing (T). In: International Conference on Automated Software Engineering (ASE), IEEE Computer Society, pp 212–223, DOI 10.1109/ASE.2015.49
2015
Cited alongside, same era.

Then

Chen Z, Ma W, Lin W, Chen L, Li Y, Xu B (2018) A study on the changes of dynamic feature code when fixing bugs: Towards the benefits and costs of python dynamic features. SCIENCE CHINA Information Sciences 61(1):012107:1–012107:18, DOI 10.1007/s11432-017-9153-3
2018
Later among the works it cites.
Holmes J, Groce A, Pinto J, Mittal P, Azimi P, Kellar K, O’Brien J (2018) TSTL: the template scripting testing language. International Journal on Software Tools for Technology Transfer 20(1):57–78, DOI 10.1007/s10009-016-0445-y
2018
Later among the works it cites.
Panichella A, Kifetew FM, Tonella P (2018b) A large scale empirical comparison of state-of-the-art search-based test case generators. Information & Software Technology 104:236–256, DOI 10.1016/j.infsof.2018.08.009
2018
Later among the works it cites.
Papadakis M, Shin D, Yoo S, Bae D (2018) Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. In: International Conference on Software Engineering (ICSE), ACM, pp 537–548, DOI 10.1145/3180155.3180183
2018
Later among the works it cites.
Selakovic M, Pradel M, Karim R, Tip F (2018) Test generation for higher-order functions in dynamic languages. Proceedings of the ACM on Programming Languages 2(OOPSLA):161:1–161:27, DOI 10.1145/3276531
2018
Later among the works it cites.
Fan Z (2019) A systematic evaluation of problematic tests generated by EvoSuite. In: International Conference on Software Engineering Companion (ICSE Companion), IEEE/ACM, pp 165–167, DOI 10.1109/ICSE-Companion.2019.00068
2019
Later among the works it cites.
MacIver D, Hatfield-Dodds Z (2019) Hypothesis: A new approach to property-based testing. Journal of Open Source Software 4(43):1891, DOI 10.21105/joss.01891
2019
Later among the works it cites.
Lukasczyk S, Kroiß F, Fraser G (2020) Automated unit test generation for python. In: International Symposium on Search Based Software Engineering (SSBSE), Springer, Lecture Notes in Computer Science, vol 12420, pp 9–24, DOI 10.1007/978-3-030-59762-7\_2
2020
Later among the works it cites.
MacIver D, Donaldson AF (2020) Test-case reduction via test-case generation: Insights from the hypothesis reducer (tool insights paper). In: European Conference on Object-Oriented Programming (ECOOP), Schloss Dagstuhl – Leibnitz-Zentrum für Informatik, Leibnitz International Proceedings in Informatics (LIPIcs), vol 166, pp 13:1–13:27, DOI 10.4230/LIPIcs.ECOOP.2020.13
2020
Later among the works it cites.
Rak-amnouykit I, McCrevan D, Milanova AL, Hirzel M, Dolby J (2020) Python 3 types in the wild: A tale of two type systems. In: ACM SIGPLAN International Symposium on Dynamic Languages (DLS), ACM, pp 57–70, DOI 10.1145/3426422.3426981
2020
Later among the works it cites.
Widyasari R, Sim SQ, Lok C, Qi H, Phan J, Tay Q, Tan C, Wee F, Tan JE, Yieh Y, Goh B, Thung F, Kang HJ, Hoang T, Lo D, Ouh EL (2020) BugsInPy: A database of existing bugs in Python programs to enable controlled testing and debugging studies. In: Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE), ACM, pp 1556–1560, DOI 10.1145/3368089.3417943
2020
Later among the works it cites.
Gruber M, Lukasczyk S, Kroiß F, Fraser G (2021) An empirical study of flaky tests in python. In: International Conference on Software Testing, Verification and Validation (ICST), IEEE, pp 148–158, DOI 10.1109/ICST49551.2021.00026
2021
Closest in time.
Mir AM, Latoskinas E, Gousios G (2021) Manytypes4py: A benchmark python dataset for machine learning-based type inference. In: IEEE Working Conference on Mining Software Repositories (MSR), IEEE, pp 585–589, DOI 10.1109/MSR52588.2021.00079
2021
Closest in time.
Schoofs E, Abdi M, Demeyer S (2021) Ampyfier: Test amplification in python. CoRR abs/2112.11155, 2112.11155
Original
2021
Closest in time.
Abdi M, Rocha H, Demeyer S, Bergel A (2022) Small-amp: Test amplification in a dynamically typed language. Empirical Software Engineering 27(128), DOI 10.1007/s10664-022-10169-8
2022
Closest in time.
Lukasczyk S (2022) Artefact to our paper “an empirical study of automated unit test generation for python”. DOI 10.5281/zenodo.6838658
2022
Closest in time.
Lukasczyk S, Fraser G (2022) Pynguin: Automated unit test generation for python. In: International Conference on Software Engineering Companion (ICSE Companion), ACM, pp 168–172, DOI 10.1145/3510454.3516829
2022
Closest in time.
Lukasczyk S, Kroiß F, Fraser G, Contributors P (2022) se2p/pynguin: Pynguin 0.25.2. DOI 10.5281/zenodo.6836225
2022
Closest in time.
Parry O, Kapfhammer GM, Hilton M, McMinn P (2022) A survey of flaky tests. ACM Transactions on Software Engineering and Methodology 31(1):17:1–17:74, DOI 10.1145/3476105
2022
Closest in time.
Trübenbach D, Müller S, Grunske L (2022) A comparative evaluation on the quality of manual and automatic test case generation techniques for scientific software—a case study of a python project for material science workflows. In: International Workshop on Search-Based Software Testing (SBST@ICSE)
2022
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…