BenchAgents: Multi-Agent Systems for Structured Benchmark Creation

Phyre: A new benchmark for physical reasoning

Original

Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019 · 1908

Earlier work this paper cites.

Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2016 · 1997

Earlier work this paper cites.

Craft: A benchmark for causal reasoning about forces and interactions

Original

Tayfun Ates, M. Samil Atesoglu, Cagatay Yigit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, and Deniz Yuret. 2020 · 2012

Earlier work this paper cites.

Vqa: Visual question answering

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015 · 2015

Earlier work this paper cites.

Clevrer: Collision events for video representation and reasoning

Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2020 · 2020

Earlier work this paper cites.

Collie: Systematic construction of constrained text generation tasks

Original

Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik Narasimhan. 2023 · 2023

Earlier work this paper cites.

Instruction-following evaluation for large language models

Original

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023 · 2023

Earlier work this paper cites.

Kitab: Evaluating llms on constraint satisfaction for information retrieval

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Ghosh Peshawaria, Ranjita Naik, and Besmira Nushi. 2024 · 2024

Earlier work this paper cites.

Claude 3.5 sonnet

Anthropic. 2024 · 2024

Earlier work this paper cites.

Assessing and verifying task utility in llm-powered applications

Negar Arabzadeh, Siqing Huo, Nikhil Mehta, Qinqyun Wu, Chi Wang, Ahmed Awadallah, Charles L. A. Clarke, and Julia Kiseleva. 2024 · 2024

Cited alongside, same era.

Eureka: Evaluating and understanding large foundation models

Original

Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, and Safoora Yousefi. 2024 · 2024

Cited alongside, same era.

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024 · 2024

Cited alongside, same era.

Autobencher: Creating salient, novel, difficult datasets for language models

Original

Xiang Lisa Li, Evan Zheran Liu, Percy Liang, and Tatsunori Hashimoto. 2024 · 2024

Cited alongside, same era.

Mistral large 2

Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation

Original

Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. 2024 · 2024

Closest in time.

S-eval: Automatic and adaptive test generation for benchmarking safety evaluation of large language models

Original

Xiaohan Yuan, Jinfeng Li, Dongxia Wang, Yuefeng Chen, Xiaofeng Mao, Longtao Huang, Hui Xue, Wenhai Wang, Kui Ren, and Jingyi Wang. 2024 · 2024

Closest in time.

Attention satisfies: A constraint-satisfaction lens on factual errors of language models

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. 2024 · 2024

Closest in time.

Natural plan: Benchmarking llms on natural language planning

Original

Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, and Denny Zhou. 2024 · 2024

Closest in time.