2020

What It Would Take to Use Mutation Testing in Industry--A Study at Facebook

Beller, Moritz, Wong, Chu-Pan, Bader, Johannes et al.

Understand

Traditionally, mutation testing generates an abundance of small deviations of a program, called mutants.

At industrial systems the scale and size of Facebook's, doing this is infeasible.
We should not create mutants that the test suite would likely fail on or that give no actionable signal to developers.
To tackle this problem, in this paper, we semi-automatically learn error-inducing patterns from a corpus of common Java coding errors and from changes that caused operational anomalies at Facebook specifically.

R. G. Hamlet, “Testing programs with the aid of a compiler,”
1977
Earlier work this paper cites.
R. A. DeMillo, R. J. Lipton, and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,”
1978
Earlier work this paper cites.
J. L. Fleiss, B. Levin, M. C. Paik
1981
Earlier work this paper cites.
W. E. Wong and A. P. Mathur, “Reducing the cost of mutation testing: An empirical study,”
1995
Earlier work this paper cites.
P. G. Frankl, S. N. Weiss, and C. Hu, “All-uses vs mutation testing: an experimental comparison of effectiveness,”
1997
Earlier work this paper cites.
F. I. Vokolos and P. G. Frankl, “Empirical evaluation of the textual differencing regression testing technique,” in
1998
Earlier work this paper cites.
A. J. Offutt and R. H. Untch, “Mutation 2000: Uniting the orthogonal,” in
2001
Earlier work this paper cites.
E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi, “Toward the determination of sufficient mutant operators for c,”
2001
Earlier work this paper cites.
I. Neamtiu, J. S. Foster, and M. Hicks, “Understanding source code evolution using abstract syntax tree matching,” in
2005
Earlier work this paper cites.
B. H. Smith and L. Williams, “On guiding the augmentation of an automated test suite via mutation analysis,”
2009
Earlier work this paper cites.
M. P. Usaola and P. R. Mateo, “Mutation testing cost reduction techniques: a survey,”
2010
Earlier work this paper cites.

Z. Ahmed, M. Zahoor, and I. Younas, “Mutation operators for object-oriented systems: A survey,” in
2010
Cited alongside, same era.
Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,”
2011
Cited alongside, same era.
G. Fraser and A. Zeller, “Mutation-driven generation of unit tests and oracles,”
2011
Cited alongside, same era.
M. A. Cachia, M. Micallef, and C. Colombo, “Towards incremental mutation testing,”
2013
Cited alongside, same era.
R. Gopinath, C. Jensen, and A. Groce, “Code coverage for suite evaluation by developers,” in
2014
Cited alongside, same era.
L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” in
2014
Cited alongside, same era.
R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, “Are mutants a valid substitute for real faults in software testing?” in
2014
Cited alongside, same era.
A. Spillner, T. Linz, and H. Schaefer,
2014
Cited alongside, same era.
R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in
2014
Cited alongside, same era.
H. Osman, M. Lungu, and O. Nierstrasz, “Mining frequent bug-fix code changes,” in
2014
Cited alongside, same era.
M. Beller, R. Bholanath, S. McIntosh, and A. Zaidman, “Analyzing the state of static analysis: A large-scale evaluation in open source software,” in
2016
Cited alongside, same era.

J. Micco, “Flaky tests at google and how we mitigate them,” 2016,
2016
Later among the works it cites.
R. Just, B. Kurtz, and P. Ammann, “Inferring mutant utility from program context,” in
2017
Later among the works it cites.
D. B. Brown, M. Vaughn, B. Liblit, and T. Reps, “The care and feeding of wild-caught mutants,” in
2017
Later among the works it cites.
G. Petrović and M. Ivanković, “State of mutation testing at google,” in
2018
Later among the works it cites.
N. Alshahwan, X. Gao, M. Harman, Y. Jia, K. Mao, A. Mols, T. Tei, and I. Zorin, “Deploying search based software engineering with sapienz at facebook,” in
2018
Later among the works it cites.
J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “Deflaker: Automatically detecting flaky tests,” in
2018
Later among the works it cites.
M. Papadakis, M. Kintis, J. Zhang, Y. Jia, Y. Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” in
2019
Later among the works it cites.
J. Bader, A. Scott, M. Pradel, and S. Chandra, “Getafix: Learning to fix bugs automatically,”
2019
Later among the works it cites.
M. Machalica, A. Samylkin, M. Porth, and S. Chandra, “Predictive test selection,” in
2019
Later among the works it cites.
P. Rijo, “An intro to mutation testing - or why coverage sucks,” 2020,
2020
Closest in time.
I. Phacility, “Phabricator,” 2020,
2020
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…