AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

Computing inter-rater reliability and its variance in the presence of high agreement

Kilem Li Gwet. 2008 · 2008

Earlier work this paper cites.

Wikidata: A free collaborative knowledgebase

Denny Vrandečić and Markus Krötzsch. 2014 · 2014

Earlier work this paper cites.

SQuAD: 100,000+ questions for machine comprehension of text

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016 · 2016

Earlier work this paper cites.

Think you have solved question answering? try arc, the ai2 reasoning challenge

Original

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018 · 2018

Earlier work this paper cites.

The narrativeqa reading comprehension challenge

Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018 · 2018

Earlier work this paper cites.

Hotpotqa: A dataset for diverse, explainable multi-hop question answering

Original

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018 · 2018

Earlier work this paper cites.

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019 · 2019

Earlier work this paper cites.

Short text topic modeling with topic distribution quantization and negative sampling decoder

Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. 2020 · 2020

Earlier work this paper cites.

Training verifiers to solve math word problems

Original

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 · 2021

Earlier work this paper cites.

Measuring massive multitask language understanding

Original

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 · 2021

Earlier work this paper cites.

Dynabench: Rethinking benchmarking in nlp

Original

Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. 2021 · 2021

Earlier work this paper cites.

Dynasent: A dynamic benchmark for sentiment analysis

Christopher Potts, Zhengxuan Wu, Atticus Geiger, and Douwe Kiela. 2021 · 2021

Earlier work this paper cites.

Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models

Original

Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, D’Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. 2022 · 2022

Earlier work this paper cites.

Data contamination: From memorization to exploitation

Original

Inbal Magar and Roy Schwartz. 2022 · 2022

Earlier work this paper cites.

Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

Original

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022 · 2022

Earlier work this paper cites.

Dynatask: A framework for creating dynamic ai benchmark tasks

Tristan Thrush, Kushal Tirumala, Anmol Gupta, Max Bartolo, Pedro Rodriguez, Tariq Kane, William Gaviria Rojas, Peter Mattson, Adina Williams, and Douwe Kiela. 2022 · 2022

Earlier work this paper cites.

Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning

Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. 2022 · 2022

Earlier work this paper cites.

Gpt-4 technical report

Original

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023 · 2023

Earlier work this paper cites.

Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023 · 2023

Earlier work this paper cites.

Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks

Original

Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023 · 2023

Earlier work this paper cites.

Mistral 7b

Original

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 · 2023

Cited alongside, same era.

Realtime qa: what’s the answer right now?

Original

Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. 2023 · 2023

Cited alongside, same era.

How long can open-source llms truly promise on context length?

Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023 · 2023

Cited alongside, same era.

An open source data contamination report for llama series models

Original

Yucheng Li. 2023 · 2023

Cited alongside, same era.

Proving test set contamination in black-box language models

The llama 3 herd of models

Original

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024 · 2024

Closest in time.

Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation

Original

Zhouhong Gu, Xiaoxuan Zhu, Haoning Ye, Lin Zhang, Jianchen Wang, Yixin Zhu, Sihang Jiang, Zhuozhi Xiong, Zihan Li, Weijie Wu, et al. 2024 · 2024

Closest in time.

Livecodebench: Holistic and contamination free evaluation of large language models for code

Original

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024 · 2024

Closest in time.

Investigating data contamination for pre-training language models

Original

Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. 2024 · 2024

Closest in time.