Fetching the paper…
Reading the bibliography…
Recent research has generated hope that inference scaling, such as resampling solutions until they pass verifiers like unit tests, could allow weaker models to match stronger ones.
Unit Test Case Generation with Transformers and Focal Context, May 2021
Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan · 2009
Earlier work this paper cites.
Avoiding Side Effects By Considering Future Tasks, October 2020
Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg · 2010
Earlier work this paper cites.
Concrete Problems in AI Safety, July 2016
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané · 2016
Earlier work this paper cites.
Program Synthesis
Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh · 2017
Earlier work this paper cites.
Program Synthesis with Large Language Models, August 2021
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton · 2021
Earlier work this paper cites.
Evaluating Large Language Models Trained on Code, July 2021
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba · 2021
Earlier work this paper cites.
Training Verifiers to Solve Math Word Problems, November 2021
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman · 2021
Earlier work this paper cites.
Measuring Coding Challenge Competence With APPS, November 2021
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt · 2021
Earlier work this paper cites.
CodeT: Code Generation with Generated Tests, November 2022
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen · 2022
Earlier work this paper cites.
Pynguin: automated unit test generation for Python
Stephan Lukasczyk and Gordon Fraser · 2022
Earlier work this paper cites.
Developers talking about code quality
Jürgen Börstler, Kwabena E. Bennin, Sara Hooshangi, Johan Jeuring, Hieke Keuning, Carsten Kleiner, Bonnie MacKellar, Rodrigo Duran, Harald Störrle, Daniel Toll, and Jelle van Assema · 2023
Earlier work this paper cites.
Chain-of-Verification Reduces Hallucination in Large Language Models, September 2023
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston · 2023
Earlier work this paper cites.
Baldur: Whole-Proof Generation and Repair with Large Language Models, March 2023
Emily First, Markus N. Rabe, Talia Ringer, and Yuriy Brun · 2023
Earlier work this paper cites.
LLM-Assisted Code Cleaning For Training Accurate Code Generators, November 2023
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E. Gonzalez, Koushik Sen, and Ion Stoica · 2023
Earlier work this paper cites.
Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction, July 2023
Sungmin Kang, Juyeon Yoon, and Shin Yoo · 2023
Earlier work this paper cites.
Large Language Models are Zero-Shot Reasoners, January 2023
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa · 2023
Earlier work this paper cites.
Holistic Evaluation of Language Models, October 2023
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda · 2023
Earlier work this paper cites.
Let’s Verify Step by Step, May 2023
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe · 2023
Earlier work this paper cites.
Self-Refine: Iterative Refinement with Self-Feedback, May 2023
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark · 2023
Earlier work this paper cites.
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis, February 2023
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong · 2023
Earlier work this paper cites.
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip · 2023
Earlier work this paper cites.
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning, June 2023
Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang · 2023
Cited alongside, same era.
Reflexion: language agents with verbal reinforcement learning
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao · 2023
Cited alongside, same era.
Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati · 2023
Cited alongside, same era.
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2023
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou · 2023
Cited alongside, same era.
LeanDojo: Theorem Proving with Retrieval-Augmented Language Models, October 2023
Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz · 2024
Closest in time.
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, March 2024
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky · 2024
Closest in time.
Code Llama: Open Foundation Models for Code, January 2024
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve · 2024
Closest in time.
Archon: An Architecture Search Framework for Inference-Time Techniques, October 2024
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan Prenger, and Anima Anandkumar · 2023
Cited alongside, same era.
Tree of Thoughts: Deliberate Problem Solving with Large Language Models, December 2023
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan · 2023
Cited alongside, same era.
Automated Unit Test Improvement using Large Language Models at Meta, February 2024
Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang · 2024
Cited alongside, same era.
Llemma: An Open Language Model For Mathematics, March 2024
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck · 2024
Cited alongside, same era.
DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning, June 2024
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar · 2024
Cited alongside, same era.
Lessons from the Trenches on Reproducible Evaluation of Language Models, May 2024
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou · 2024
Cited alongside, same era.
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, September 2024
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini · 2024
Cited alongside, same era.
ChatUniTest: A Framework for LLM-Based Test Generation
Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin · 2024
Cited alongside, same era.
Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, and Azalia Mirhoseini · 2024
Closest in time.
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning, October 2024
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar · 2024
Closest in time.
Using Large Language Models to Generate JUnit Tests: An Empirical Study, March 2024
Mohammed Latif Siddiq, Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes · 2024
Closest in time.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar · 2024
Closest in time.
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism, July 2024
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin · 2024
Closest in time.
Chain of Thoughtlessness? An Analysis of CoT in Planning, June 2024
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati · 2024
Closest in time.
An In-Context Learning Agent for Formal Theorem-Proving, August 2024
Amitayush Thakur, George Tsoukalas, Yeming Wen, Jimmy Xin, and Swarat Chaudhuri · 2024
Closest in time.
LLMs cannot find reasoning errors, but can correct them given the error location, June 2024
Gladys Tyen, Hassan Mansoor, Victor Cărbune, Peter Chen, and Tony Mak · 2024
Closest in time.
General Purpose Verification for Chain of Thought Prompting, April 2024
Robert Vacareanu, Anurag Pratik, Evangelia Spiliopoulou, Zheng Qi, Giovanni Paolini, Neha Anna John, Jie Ma, Yassine Benajiba, and Miguel Ballesteros · 2024
Closest in time.
From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models, June 2024
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui · 2024
Closest in time.
Thinking LLMs: General Instruction Following with Thought Generation, October 2024
Tianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar · 2024
Closest in time.
DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data, May 2024
Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang · 2024
Closest in time.
Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement, June 2024
Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang · 2024
Closest in time.
tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, June 2024
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan · 2024
Closest in time.
Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun · 2024
Closest in time.
Li Zhong, Zilong Wang, and Jingbo Shang · 2024
Closest in time.
Agent-as-a-Judge: Evaluate Agents with Agents, October 2024
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber · 2024
Closest in time.