2023

FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair

Fatima, Sakina, Hemmati, Hadi, Briand, Lionel

Understand

Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test, causing confusion and wasting development effort.

While machine learning models have been used to predict flakiness and its root causes, there is much less work on providing support to fix the problem.
To address this gap, in this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis.
We do this for a subset of flaky tests where the root cause of flakiness is in the test itself and not in the production code.

Built on

Bootstrapping: A nonparametric approach to statistical inference
Christopher Z Mooney, Robert D Duval, and Robert Duvall · 1993
Earlier work this paper cites.
An exact test for population differentiation
Michel Raymond and François Rousset · 1995
Earlier work this paper cites.
Confidence intervals for measures of interaction
Susan F Assmann, David W Hosmer, Stanley Lemeshow, and Kenneth A Mundt · 1996
Earlier work this paper cites.
Learning string-edit distance
Eric Sven Ristad and Peter N Yianilos · 1998
Earlier work this paper cites.
Bleu: a method for automatic evaluation of machine translation
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu · 2002
Earlier work this paper cites.
xUnit test patterns: Refactoring test code
Gerard Meszaros · 2007
Earlier work this paper cites.
Wilcoxon signed-rank test
Robert F Woolson · 2007
Earlier work this paper cites.
Natural language processing (almost) from scratch
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa · 2011
Earlier work this paper cites.
Interrater reliability: the kappa statistic
Mary L McHugh · 2012
Earlier work this paper cites.
Automatic defect categorization
Ferdian Thung, David Lo, and Lingxiao Jiang · 2012
Earlier work this paper cites.
The Art of Unit Testing: with examples in C
Roy Osherove · 2013
Earlier work this paper cites.
On the difficulty of training recurrent neural networks
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio · 2013
Earlier work this paper cites.
Applied logistic regression
David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant · 2013
Earlier work this paper cites.
The chi-square test of independence
Mary L McHugh · 2013
Earlier work this paper cites.
An empirical analysis of flaky tests
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov · 2014
Earlier work this paper cites.
Neural machine translation of rare words with subword units
Original
Rico Sennrich, Barry Haddow, and Alexandra Birch · 2015
Earlier work this paper cites.
Facenet: A unified embedding for face recognition and clustering
Florian Schroff, Dmitry Kalenichenko, and James Philbin · 2015
Earlier work this paper cites.
Siamese neural networks for one-shot image recognition. in icml deep learning workshop (vol. 2)
G Koch, R Zemel, and R Salakhutdinov · 2015
Earlier work this paper cites.
Learning text similarity with siamese recurrent networks
Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru · 2016
Earlier work this paper cites.
Siamese recurrent architectures for learning sentence similarity
Jonas Mueller and Aditya Thyagarajan · 2016
Earlier work this paper cites.
A survey of predictive modeling on imbalanced domains
Paula Branco, Luís Torgo, and Rita P Ribeiro · 2016
Earlier work this paper cites.
Flaky tests at google and how we mitigate them—googblogs. com, 2016
J Micco · 2016
Earlier work this paper cites.
Deflaker: Automatically detecting flaky tests
Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov · 2018
Earlier work this paper cites.
A twofold siamese network for real-time object tracking
Anfeng He, Chong Luo, Xinmei Tian, and Wenjun Zeng · 2018
Earlier work this paper cites.
Deep learning using rectified linear units (relu)
Original
Abien Fred Agarap · 2018
Earlier work this paper cites.
Smells in software test code: A survey of knowledge in industry and academia
Vahid Garousi and Barış Küçük · 2018
Earlier work this paper cites.
How we approach testing vsts to enable continuous delivery, 2019
Brian Harry · 2019
Earlier work this paper cites.
Flakime: Laboratory-controlled test flakiness impact assessment. a case study on mutation testing and program repair
Original
Maxime Cordy, Renaud Rwemalika, Mike Papadakis, and Mark Harman · 2019
Earlier work this paper cites.
idflakies: A framework for detecting and partially classifying flaky tests
Wing Lam, Reed Oei, August Shi, Darko Marinov, and Tao Xie · 2019
Earlier work this paper cites.
ifixflakies: A framework for automatically fixing order-dependent flaky tests
August Shi, Wing Lam, Reed Oei, Tao Xie, and Darko Marinov · 2019
Earlier work this paper cites.
Understanding flaky tests: The developer’s perspective
Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli · 2019
Earlier work this paper cites.

Similar

Automated program repair
Claire Le Goues, Michael Pradel, and Abhik Roychoudhury · 2019
Cited alongside, same era.
Root causing flaky tests in a large-scale industrial setting
Wing Lam, Patrice Godefroid, Suman Nath, Anirudh Santhiar, and Suresh Thummalapenta · 2019
Cited alongside, same era.
A large-scale longitudinal study of flaky tests
Wing Lam, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, and Jonathan Bell · 2020
Cited alongside, same era.
What is the vocabulary of flaky tests?
Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo d’Amorim, Christoph Treude, and Antonia Bertolino · 2020
Cited alongside, same era.
An evaluation of machine learning methods for predicting flaky tests
Azeem Ahmad · 2020
Cited alongside, same era.
International Dataset of Flaky Tests (IDoFT), 2020
Wing Lam · 2020
Cited alongside, same era.
A study on the lifecycle of flaky tests
Wing Lam, Kıvanç Muşlu, Hitesh Sajnani, and Suresh Thummalapenta · 2020
Cited alongside, same era.
Codebert: A pre-trained model for programming and natural languages
Original
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al · 2020
Cited alongside, same era.
Generalizing from a few examples: A survey on few-shot learning
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni · 2020
Cited alongside, same era.
Understanding reproducibility and characteristics of flaky tests through test reruns in java projects
Wing Lam, Stefan Winter, Angello Astorga, Victoria Stodden, and Darko Marinov · 2020
Cited alongside, same era.
Dependent-test-aware regression testing techniques
Wing Lam, August Shi, Reed Oei, Sai Zhang, Michael D Ernst, and Tao Xie · 2020
Cited alongside, same era.
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al · 2020
Cited alongside, same era.

Then

Peeler: Learning to effectively predict flakiness without running tests
Yihao Qin, Shangwen Wang, Kui Liu, Bo Lin, Hongjun Wu, Li Li, Xiaoguang Mao, and Tegawendé F Bissyandé · 2022
Later among the works it cites.
Flakify: A black-box, language model-based predictor for flaky tests
Sakina Fatima, Taher A Ghaleb, and Lionel Briand · 2022
Later among the works it cites.
What do developer-repaired flaky tests tell us about the effectiveness of automated flaky test detection?
Owain Parry, Gregory M Kapfhammer, Michael Hilton, and Phil McMinn · 2022
Later among the works it cites.
An empirical study of flaky tests in javascript
Negar Hashemi, Amjed Tahir, and Shawn Rasheed · 2022
Later among the works it cites.
Unixcoder: Unified cross-modal pre-training for code representation
Original
Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin · 2022
Later among the works it cites.
Reasoning with language model prompting: A survey
Original
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen · 2022
Later among the works it cites.
ipflakies: A framework for detecting and fixing python order-dependent flaky tests
Ruixin Wang, Yang Chen, and Wing Lam · 2022
Later among the works it cites.
A survey on how test flakiness affects developers and what support they need to address it
Martin Gruber and Gordon Fraser · 2022
Later among the works it cites.
An empirical evaluation of github copilot’s code suggestions
Nhan Nguyen and Sarah Nadi · 2022
Later among the works it cites.
Can openai’s codex fix bugs? an evaluation on quixbugs
Julian Aron Prenner, Hlib Babii, and Romain Robbes · 2022
Later among the works it cites.
Less training, more repairing please: revisiting automated program repair via zero-shot learning
Chunqiu Steven Xia and Lingming Zhang · 2022
Later among the works it cites.
Practical flaky test prediction using common code evolution and test history data
Martin Gruber, Michael Heine, Norbert Oster, Michael Philippsen, and Gordon Fraser · 2023
Closest in time.
Flakycat: Predicting flaky tests categories using few-shot learning
Amal Akli, Guillaume Haben, Sarra Habchi, Mike Papadakis, and Yves Le Traon · 2023
Closest in time.
Gpt-4 technical report. arxiv 2303.08774
R OpenAI · 2023
Closest in time.
Codescore: Evaluating code generation by learning code execution
Original
Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin · 2023
Closest in time.
Traf: Time-based repair for asynchronous wait flaky tests in web testing
Original
Yu Pei, Jeongju Sohn, Sarra Habchi, and Mike Papadakis · 2023
Closest in time.
An empirical comparison of pre-trained models of source code
Original
Changan Niu, Chuanyi Li, Vincent Ng, Dongxiao Chen, Jidong Ge, and Bin Luo · 2023
Closest in time.
A survey of large language models
Original
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al · 2023
Closest in time.
A survey of large language models for code: Evolution, benchmarking, and future trends
Original
Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen · 2023
Closest in time.
Large language models for software engineering: Survey and open problems
Original
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang · 2023
Closest in time.
Examining zero-shot vulnerability repair with large language models
Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt · 2023
Closest in time.
Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design
Original
Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt · 2023
Closest in time.
A critical review of large language model on software engineering: An example from chatgpt and automated program repair
Original
Quanjun Zhang, Tongke Zhang, Juan Zhai, Chunrong Fang, Bowen Yu, Weisong Sun, and Zhenyu Chen · 2023
Closest in time.
Chatgpt for vulnerability detection, classification, and repair: How far are we?
Original
Michael Fu, Chakkrit Tantithamthavorn, Van Nguyen, and Trung Le · 2023
Closest in time.
Openai platform documentation: Continuous model upgrades
OpenAI · 2023
Closest in time.
Automatic software code repair using deep learning techniques
Mohammad Mahdi Abdollahpour, Mehrdad Ashtiani, and Fatemeh Bakhshi · 2023
Closest in time.
Improving chatgpt prompt for code generation
Original
Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan · 2023
Closest in time.
Out of the bleu: how should we assess quality of the code generation models?
Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin · 2023
Closest in time.
Gpt-3-powered type error debugging: Investigating the use of large language models for code repair
Francisco Ribeiro, José Nuno Castro de Macedo, Kanae Tsushima, Rui Abreu, and João Saraiva · 2023
Closest in time.
Automated program repair in the era of large pre-trained language models
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang · 2023
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…