None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks

Understanding foundation models: Are we back in 1924?

Original

Alan F. Smeaton. 2024 · 1924

Earlier work this paper cites.

Scaling laws for neural language models

Original

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020 · 2001

Earlier work this paper cites.

Language models are few-shot learners

Original

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, and Pranav Shyam et al. 2020 · 2005

Earlier work this paper cites.

Measuring Massive Multitask Language Understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021a · 2009

Earlier work this paper cites.

Training Verifiers to Solve Math Word Problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Impact of pretraining term frequencies on few-shot numerical reasoning

Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022 · 2022

Earlier work this paper cites.

MEGA: Multilingual Evaluation of Generative AI

Original

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023 · 2023

Earlier work this paper cites.

Faith and fate: Limits of transformers on compositionality

Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023 · 2023

Earlier work this paper cites.

Mistral 7B

Original

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, and Florian others Bressand. 2023 · 2023

Earlier work this paper cites.

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Original

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023 · 2023

Earlier work this paper cites.

NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark

Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023 · 2023

Earlier work this paper cites.

LLaMA: Open and Efficient Foundation Language Models

Original

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, and Lachaux et al. 2023 · 2023

Earlier work this paper cites.

On the robustness of chatgpt: An adversarial and out-of-distribution perspective

Original

Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. 2023 · 2023

Earlier work this paper cites.

M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

Original

Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023 · 2023

Earlier work this paper cites.

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Original

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023 · 2023

Cited alongside, same era.

PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

Original

Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Zhenqiang Gong, and Xing Xie. 2023 · 2023

Cited alongside, same era.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, and Aleman et al. 2024 · 2024

Cited alongside, same era.

The Claude 3 Model Family: Opus, Sonnet, Haiku

Anthropic. 2024 · 2024

Cited alongside, same era.

Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source LLMs

Introducing meta llama 3: The most capable openly available llm to date

Meta. 2024 · 2024

Later among the works it cites.

Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

Original

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024 · 2024

Later among the works it cites.

Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Models

Original

Marianna Nezhurina, Lucia Cipolina-Kun, Mehdi Cherti, and Jenia Jitsev. 2024 · 2024

Later among the works it cites.

Arithmetic without algorithms: Language models solve math with a bag of heuristics

Original

Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov. 2024 · 2024

Later among the works it cites.

Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning