Test Oracle Automation in the era of LLMs

BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

Original

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019 · 1910

Earlier work this paper cites.

Applying "Design by Contract"

Bertrand Meyer. 1992 · 1992

Earlier work this paper cites.

Metamorphic testing: a new approach for generating next test cases. In Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998

Tsong Y Chen, Shing C Cheung, and Shiu Ming Yiu. 1998 · 1998

Earlier work this paper cites.

Beyond Assertions: Advanced Specification and Verification with JML and ESC/Java2. In Formal Methods for Components and Objects, 4th International Symposium, FMCO 2005, Amsterdam, The Netherlands, November 1-4, 2005, Revised Lectures . 342–363

Patrice Chalin, Joseph R. Kiniry, Gary T. Leavens, and Erik Poll. 2005 · 2005

Earlier work this paper cites.

The Daikon system for dynamic detection of likely invariants

Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007 · 2007

Earlier work this paper cites.

Mutation-driven generation of unit tests and oracles. In ISSTA . ACM, 147–158

Gordon Fraser and Andreas Zeller. 2010 · 2010

Earlier work this paper cites.

Inferring better contracts. In Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu , HI, USA, May 21-28, 2011 , Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.). ACM, 191–200

Yi Wei, Carlo A. Furia, Nikolay Kazmin, and Bertrand Meyer. 2011 · 2011

Earlier work this paper cites.

The Oracle Problem in Software Testing: A Survey

Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015 · 2014

Earlier work this paper cites.

Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014 , Corina S. Pasareanu and Darko Marinov (Eds.). ACM, 437–440

René Just, Darioush Jalali, and Michael D. Ernst. 2014 · 2014

Earlier work this paper cites.

Metamorphic Testing for Software Quality Assessment: A Study of Search Engines

Zhiquan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2016 · 2015

Earlier work this paper cites.

Test oracle assessment and improvement. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, Saarbrücken, Germany, July 18-20, 2016 , Andreas Zeller and Abhik Roychoudhury (Eds.). ACM, 247–258

Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016 · 2016

Earlier work this paper cites.

A Survey on Metamorphic Testing

Sergio Segura, Gordon Fraser, Ana Belén Sánchez, and Antonio Ruiz Cortés. 2016 · 2016

Earlier work this paper cites.

Metamorphic Testing of RESTful Web APIs

Sergio Segura, José Antonio Parejo, Javier Troya, and Antonio Ruiz Cortés. 2018 · 2017

Earlier work this paper cites.

Metamorphic Testing: Testing the Untestable

Sergio Segura, Dave Towey, Zhi Quan Zhou, and Tsong Yueh Chen. 2020 · 2018

Earlier work this paper cites.

Evolutionary Improvement of Assertion Oracles. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020) . Association for Computing Machinery, New York, NY, USA, 1178–1189

Valerio Terragni, Gunel Jahangirova, Paolo Tonella, and Mauro Pezzè. 2020 · 2020

Cited alongside, same era.

Synthesizing contracts correct modulo a test generator

Angello Astorga, Shambwaditya Saha, Ahmad Dinkins, Felicia Wang, P. Madhusudan, and Tao Xie. 2021 · 2021

Cited alongside, same era.

MeMo: Automatically identifying metamorphic relations in Javadoc comments for test automation

Arianna Blasi, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Antonio Carzaniga. 2021 · 2021

Cited alongside, same era.

Evaluating Large Language Models Trained on Code

Original

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman et al. 2021 · 2021

Cited alongside, same era.

Impact of Code Language Models on Automated Program Repair. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 1430–1442

Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023 · 2023

Later among the works it cites.

CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 919–931

Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023 · 2023

Later among the works it cites.

Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2450–2462

Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023 · 2023

Later among the works it cites.

Learning Deep Semantics for Test Completion. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2111–2123