Built on

Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021
Earlier work this paper cites.
Highly accurate protein structure prediction with alphafold
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021 · 2021
Earlier work this paper cites.
Self-consistency improves chain of thought reasoning in language models
Original
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022 · 2022
Earlier work this paper cites.
React: Synergizing reasoning and acting in language models
Original
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022 · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Earlier work this paper cites.
Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries
Original
Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, Zejian Yuan, and Dongmei Zhang. 2023 · 2023
Earlier work this paper cites.
Benchmarking large language models as ai research agents
Original
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2023 · 2023
Earlier work this paper cites.
Ds-1000: A natural and reliable benchmark for data science code generation
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023 · 2023
Earlier work this paper cites.
Insightpilot: An llm-empowered automated data exploration system
Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. 2023 · 2023
Earlier work this paper cites.
Code llama: Open foundation models for code
Original
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023 · 2023
Earlier work this paper cites.
Gemini: a family of highly capable multimodal models
Original
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023 · 2023
Earlier work this paper cites.

Similar

Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x
Original
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023 · 2023
Cited alongside, same era.
Chatglm: A family of large language models from glm-130b to glm-4 all tools
Original
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024 · 2024
Cited alongside, same era.
Deepseek-coder: When the large language model meets programming–the rise of code intelligence
Original
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024 · 2024
Cited alongside, same era.
Data interpreter: An llm agent for data science
Original
Sirui Hong, Yizhang Lin, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Lingyao Zhang, Mingchen Zhuge, et al. 2024 · 2024
Cited alongside, same era.

Then

Starcoder 2 and the stack v2: The next generation
Original
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024 · 2024
Later among the works it cites.
The ai scientist: Towards fully automated open-ended scientific discovery
Original
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024 · 2024
Later among the works it cites.
Gemma: Open models based on gemini research and technology
Original
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024 · 2024
Later among the works it cites.
Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems
Original
Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, and Jie Tang. 2024 · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2025

DataSciBench: An LLM Agent Benchmark for Data Science

Zhang, Dan, Zhoubian, Sining, Cai, Min et al.

Understand

This paper presents DataSciBench, a comprehensive benchmark for evaluating Large Language Model (LLM) capabilities in data science.

Recent related benchmarks have primarily focused on single tasks, easily obtainable ground truth, and straightforward evaluation metrics, which limits the scope of tasks that can be evaluated.
In contrast, DataSciBench is constructed based on a more comprehensive and curated collection of natural and challenging prompts for uncertain ground truth and evaluation metrics.
We develop a semi-automated pipeline for generating ground truth (GT) and validating evaluation metrics.

Built on

Evaluating large language models trained on code
Original
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 · 2021
Earlier work this paper cites.
Highly accurate protein structure prediction with alphafold
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. 2021 · 2021
Earlier work this paper cites.
Self-consistency improves chain of thought reasoning in language models
Original
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022 · 2022
Earlier work this paper cites.
React: Synergizing reasoning and acting in language models
Original
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022 · 2022
Earlier work this paper cites.
Gpt-4 technical report
Original
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023
Earlier work this paper cites.
Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries
Original
Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, Zejian Yuan, and Dongmei Zhang. 2023 · 2023
Earlier work this paper cites.
Benchmarking large language models as ai research agents
Original
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. 2023 · 2023
Earlier work this paper cites.
Ds-1000: A natural and reliable benchmark for data science code generation
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023 · 2023
Earlier work this paper cites.
Insightpilot: An llm-empowered automated data exploration system
Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. 2023 · 2023
Earlier work this paper cites.
Code llama: Open foundation models for code
Original
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023 · 2023
Earlier work this paper cites.
Gemini: a family of highly capable multimodal models
Original
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023 · 2023
Earlier work this paper cites.

Similar

Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x
Original
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, et al. 2023 · 2023
Cited alongside, same era.
Chatglm: A family of large language models from glm-130b to glm-4 all tools
Original
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al. 2024 · 2024
Cited alongside, same era.
Deepseek-coder: When the large language model meets programming–the rise of code intelligence
Original
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024 · 2024
Cited alongside, same era.
Data interpreter: An llm agent for data science
Original
Sirui Hong, Yizhang Lin, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Lingyao Zhang, Mingchen Zhuge, et al. 2024 · 2024
Cited alongside, same era.

Then

Starcoder 2 and the stack v2: The next generation
Original
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024 · 2024
Later among the works it cites.
The ai scientist: Towards fully automated open-ended scientific discovery
Original
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024 · 2024
Later among the works it cites.
Gemma: Open models based on gemini research and technology
Original
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024 · 2024
Later among the works it cites.
Battleagentbench: A benchmark for evaluating cooperation and competition capabilities of language models in multi-agent systems
Original
Wei Wang, Dan Zhang, Tao Feng, Boyan Wang, and Jie Tang. 2024 · 2024
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Infiagent-dabench: Evaluating agents on data analysis tasks

Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, et al. 2024 · 2024

Cited alongside, same era.

Qwen2. 5-coder technical report

Original

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024 · 2024

Cited alongside, same era.

Gpt-4o system card

Original

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024 · 2024

Cited alongside, same era.

Openai o1 system card

Original

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024 · 2024

Cited alongside, same era.

Livecodebench: Holistic and contamination free evaluation of large language models for code

Original

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024 · 2024

Cited alongside, same era.

Sheetcopilot: Bringing software productivity to the next level through large language models

Hongxin Li, Jingran Su, Yuntao Chen, Qing Li, and ZHAO-XIANG ZHANG. 2024 · 2024

Cited alongside, same era.

xtrimopglm: unified 100b-scale pre-trained transformer for deciphering the language of protein

Original

Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, et al. 2024a

Cited in the paper.

Viseval: A benchmark for data visualization in the era of large language models

Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, and Yuqing Yang. 2024b

Cited in the paper.

Scenegenagent: Precise industrial scene generation with coding agent

Original

Xiao Xia, Dan Zhang, Zibo Liao, Zhenyu Hou, Tianrui Sun, Jing Li, Ling Fu, and Yuxiao Dong. 2024 · 2024