Fetching the paper…
Reading the bibliography…
Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks.
Houdini, an annotation assistant for esc/java
Cormac Flanagan and K. Rustan M. Leino · 2001
Earlier work this paper cites.
Tracking down software bugs using automatic anomaly detection
Sudheendra Hangal and Monica S. Lam · 2002
Earlier work this paper cites.
Static specification mining using automata-based abstractions
Sharon Shoham, Eran Yahav, Stephen J. Fink, and Marco Pistoia · 2007
Earlier work this paper cites.
The daikon system for dynamic detection of likely invariants
Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao · 2007
Earlier work this paper cites.
/*icomment: bugs or bad comments?*/
Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou · 2007
Earlier work this paper cites.
acomment: mining annotations from comments and code to detect interrupt related concurrency bugs
Lin Tan, Yuanyuan Zhou, and Yoann Padioleau · 2011
Earlier work this paper cites.
Inferring method specifications from natural language api descriptions
Rahul Pandita, Xusheng Xiao, Hao Zhong, Tao Xie, Stephen Oney, and Amit M. Paradkar · 2012
Earlier work this paper cites.
@tcomment: Testing javadoc comments to detect comment-code inconsistencies
Shin Hwei Tan, Darko Marinov, Lin Tan, and Gary T. Leavens · 2012
Earlier work this paper cites.
Supporting oracle construction via static analysis
Junjie Chen, Yanwei Bai, Dan Hao, Lingming Zhang, Lu Zhang, Bing Xie, and Hong Mei · 2016
Earlier work this paper cites.
Improving language understanding by generative pre-training
Alec Radford and Karthik Narasimhan · 2018
Earlier work this paper cites.
Translating code comments to procedure specifications
Arianna Blasi, Alberto Goffi, Konstantin Kuznetsov, Alessandra Gorla, Michael D. Ernst, Mauro Pezzè, and Sergio Delgado Castellanos · 2018
Earlier work this paper cites.
An introduction to software verification with whiley
David J. Pearce, Mark Utting, and Lindsay J. Groves · 2018
Earlier work this paper cites.
Cln2inv: Learning loop invariants with continuous logic networks
Gabriel Ryan, Justin Wong, Jianan Yao, Ronghui Gu, and Suman Sekhar Jana · 2019
Earlier work this paper cites.
Training binary classifiers as data structure invariants
Facundo Molina, Renzo Degiovanni, Pablo Ponzio, Germán Regis, Nazareno Aguirre, and Marcelo Fabian Frias · 2019
Earlier work this paper cites.
Generating accurate assert statements for unit test cases using pretrained transformers
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, and Neel Sundaresan · 2020
Cited alongside, same era.
Code2inv: A deep learning framework for program verification
Xujie Si, Aaditya Naik, Hanjun Dai, M. Naik, and Le Song · 2020
Cited alongside, same era.
Language models are few-shot learners
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei · 2020
Cited alongside, same era.
Evaluating large language models trained on code
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, Suchir Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021
A survey of large language models
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen · 2023
Later among the works it cites.
Starcoder: may the source be with you!
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nourhan Fahmy, Urvashi Bhattacharyya, W. Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jana Ebert, Tri Dao, Mayank Mishra, Alexander Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean M. Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries · 2023
Later among the works it cites.
Codegen2: Lessons for training llms on programming and natural languages
Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou · 2023
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi · 2021
Cited alongside, same era.
Syntactic vs. semantic similarity of artificial and real faults in mutation testing studies
Milos Ojdanic, Aayush Garg, Ahmed Khanfir, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon · 2021
Cited alongside, same era.
Tabllm: Few-shot classification of tabular data with large language models
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David A. Sontag · 2022
Cited alongside, same era.
Emergent abilities of large language models
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus · 2022
Cited alongside, same era.
Codegen: An open large language model for code with multi-turn program synthesis
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong · 2022
Cited alongside, same era.
Deep learning driven natural languages text to sql query conversion: A survey
Ayush Kumar, Parth Nagarkar, Prabhav Nalhe, and Sanjeev Vijayakumar · 2022
Cited alongside, same era.
Recent advances in text-to-sql: A survey of what we have and what we expect
Naihao Deng, Yulong Chen, and Yue Zhang · 2022
Cited alongside, same era.
Prompting large language model for machine translation: A case study
Biao Zhang, Barry Haddow, and Alexandra Birch · 2023
Cited alongside, same era.
Impact of code language models on automated program repair
Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan · 2023
Later among the works it cites.
Adaptive test generation using a large language model
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip · 2023
Later among the works it cites.
Using large language models to generate junit tests: An empirical study
Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes · 2023
Later among the works it cites.
On comparing mutation testing tools through learning-based mutant selection
Milos Ojdanic, Ahmed Khanfir, Aayush Garg, Renzo Degiovanni, Mike Papadakis, and Yves Le Traon · 2023
Later among the works it cites.
Levels of agi: Operationalizing progress on the path to agi
Meredith Ringel Morris, Jascha Narain Sohl-Dickstein, Noah Fiedel, Tris Brian Warkentin, Allan Dafoe, Aleksandra Faust, Cl’ement Farabet, and Shane Legg · 2023
Later among the works it cites.
Can large language models reason about program invariants?
Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin · 2023
Later among the works it cites.
Can chatgpt support software verification?
Christian Janssen, Cedric Richter, and Heike Wehrheim · 2023
Later among the works it cites.
Formalizing natural language intent into program specifications via large language models
Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K. Lahiri · 2023
Later among the works it cites.
https://www.ciodive.com/news/github-copilot-subscriber-count-revenue-growth/706201/
Github copilot drives revenue growth amid subscriber base expansion, 2024 · 2024
Closest in time.
Specgen: Automated generation of formal program specifications via large language models
Lezhi Ma, Shangqing Liu, Yi Li, Xiaofei Xie, and Lei Bu · 2024
Closest in time.
Gemma: Open models based on gemini research and technology
Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L’eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Am’elie H’eliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Cl’ement Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vladimir Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Brian Warkentin, Ludovic Peran, Minh Giang, Cl’ement Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy · 2024
Closest in time.