Fetching the paper…
Reading the bibliography…
We introduce xbench, a dynamic, profession-aligned evaluation suite designed to bridge the gap between AI agent capabilities and real-world productivity.
Fundamentals of item response theory , volume 2
Ronald K Hambleton, Hariharan Swaminathan, and H Jane Rogers · 1991
Earlier work this paper cites.
Scaling laws for neural language models, 2020
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2001
Earlier work this paper cites.
Constitutional ai: Harmlessness from ai feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al · 2022
Earlier work this paper cites.
Training compute-optimal large language models, 2022
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre · 2022
Earlier work this paper cites.
Opencompass: A universal evaluation platform for foundation models
OpenCompass Contributors · 2023
Earlier work this paper cites.
Mind2web: Towards a generalist agent for the web
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su · 2023
Earlier work this paper cites.
Swe-bench: Can language models resolve real-world github issues?
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan · 2023
Earlier work this paper cites.
Scaling laws for hyperparameter optimization, 2023
Arlind Kadra, Maciej Janowski, Martin Wistuba, and Josif Grabocka · 2023
Earlier work this paper cites.
Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang · 2023
Earlier work this paper cites.
Experimental evidence on the productivity effects of generative artificial intelligence
Shakked Noy and Whitney Zhang · 2023
Earlier work this paper cites.
GPQA: A graduate-level Google-proof Q&A benchmark
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman · 2023
Earlier work this paper cites.
Arc prize 2024: Technical report
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers · 2024
Earlier work this paper cites.
Autonomous medical evaluation for guideline adherence of large language models
Dennis Fast, Lisa C Adams, Felix Busch, Conor Fallon, Marc Huppertz, Robert Siepmann, Philipp Prucker, Nadine Bayerl, Daniel Truhn, Marcus Makowski, et al · 2024
Earlier work this paper cites.
Livecodebench: Holistic and contamination free evaluation of large language models for code
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica · 2024
Earlier work this paper cites.
Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al · 2024
Cited alongside, same era.
Wildbench: Benchmarking llms with challenging tasks from real users in the wild
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi · 2024
Cited alongside, same era.
Hello GPT-4o, 2024
OpenAI · 2024
Cited alongside, same era.
Webcanvas: Benchmarking web agents in online environments
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al · 2024
Cited alongside, same era.
Language agents: Foundations, prospects, and risks
Yu Su, Diyi Yang, Shunyu Yao, and Tao Yu · 2024
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al · 2025
Closest in time.
Gemini 2.5, 2025
Google DeepMind · 2025
Closest in time.
SuperGPQA: Scaling LLM evaluation across 285 graduate disciplines
Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al · 2025
Closest in time.
Scaling laws for downstream task performance in machine translation, 2025
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo · 2025
Closest in time.
Collaborating with ai agents: Field experiments on teamwork, productivity, and performance
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
MMLU-Pro: A more robust and challenging multi-task language understanding benchmark
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen · 2024
Cited alongside, same era.
Livebench: A challenging, contamination-free llm benchmark
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al · 2024
Cited alongside, same era.
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al · 2024
Cited alongside, same era.
Benchmarking benchmark leakage in large language models
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu · 2024
Cited alongside, same era.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al · 2024
Cited alongside, same era.
τ \tau -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan · 2024
Cited alongside, same era.
Claude 3.7 Sonnet, 2025
Anthropic · 2025
Cited alongside, same era.
Harang Ju and Sinan Aral · 2025
Closest in time.
The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, 2025
Meta-AI · 2025
Closest in time.
Measuring ai ability to complete long tasks
METR · 2025
Closest in time.
Introducing openai o3 and o4-mini, 2025
OpenAI · 2025
Closest in time.
Vista: Visual-language understanding leaderboard, 2025
Scale AI · 2025
Closest in time.
Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing · 2025
Closest in time.
Paperbench: Evaluating ai’s ability to replicate ai research
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al · 2025
Closest in time.
Unveiling downstream performance scaling of llms: A clustering-based perspective, 2025
Chengyin Xu, Kaiyuan Chen, Xiao Li, Ke Shen, and Chenggang Li · 2025
Closest in time.
The second half, 2025
Shunyu Yao · 2025
Closest in time.