2021

AI and the Everything in the Whole Wide World Benchmark

Raji, Inioluwa Deborah, Bender, Emily M., Paullada, Amandalynne et al.

Understand

There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks.

These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones on the path towards flexible and generalizable AI systems.
State-of-the-art performance on these benchmarks is widely understood as indicative of progress towards these long-term goals.
In this position paper, we explore the limits of such benchmarks in order to reveal the construct validity issues in their framing as the functionally "general" broad measures of progress they are set up to be.

Built on

Diversity in faces, 2019
Original
Michele Merler, Nalini Ratha, Rogerio S. Feris, and John R. Smith · 1901
Earlier work this paper cites.
Counterfactual sensitivity and robustness, 2019
Original
Timothy Christensen and Benjamin Connault · 1904
Earlier work this paper cites.
“Digital Natives”: How Medical and Indigenous Histories Matter for Big Data
Joanna Radin · 1933
Earlier work this paper cites.
Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques
Jacob Yerushalmy · 1947
Earlier work this paper cites.
An analysis of perceptual confusions among some English consonants
George A Miller and Patricia E Nicely · 1955
Earlier work this paper cites.
Report on a general problem-solving program
Allen Newell, John C. Shaw, and Herbert A. Simon · 1959
Earlier work this paper cites.
Grover and the Everything in the Whole Wide World Museum
Norman Stiles and Daniel Wilcox · 1974
Earlier work this paper cites.
A tutorial on speech understanding systems
Allen Newell · 1975
Earlier work this paper cites.
The conduit metaphor: A case of frame conflict in our language about language
Michael J Reddy · 1979
Earlier work this paper cites.
The evolution of benchmarking as a computer performance evaluation technique
Byron C Lewis and Albert E Crews · 1985
Earlier work this paper cites.
Situated knowledges: The science question in feminism and the privilege of partial perspective
Donna Haraway · 1988
Earlier work this paper cites.
Surveying
Heribert Kahmen and Wolfgang Faig · 1988
Earlier work this paper cites.
Testing heuristics: We have it all wrong
John N Hooker · 1995
Earlier work this paper cites.
WordNet: A lexical database for English
George A Miller · 1995
Earlier work this paper cites.
Using Language
Herbert H. Clark · 1996
Earlier work this paper cites.
Message understanding conference-6: A brief history
Ralph Grishman and Beth Sundheim · 1996
Earlier work this paper cites.
Artificial Knowing: Gender and the Thinking Machine
Alison Adam · 1998
Earlier work this paper cites.
Senseval: An exercise in evaluating word sense disambiguation programs
Adam Kilgarri · 1998
Earlier work this paper cites.
The empirical assessment of construct validity
Scott W O’Leary-Kelly and Robert J Vokurka · 1998
Earlier work this paper cites.
The FERET evaluation methodology for face-recognition algorithms
P Jonathon Phillips, Hyeonjoon Moon, Syed A Rizvi, and Patrick J Rauss · 2000
Earlier work this paper cites.
A hierarchy of limitations in machine learning
Original
Momin M. Malik · 2002
Earlier work this paper cites.
The problem with metrics is a fundamental problem for AI, 2020
Original
Rachel Thomas and David Uminsky · 2002
Earlier work this paper cites.
Replacing the ‘view from nowhere’: A pragmatist-feminist science classroom
Sarah M Stitzlein · 2004
Earlier work this paper cites.
Shared-task evaluations in HLT: Lessons for NLG
Anja Belz and Adam Kilgarriff · 2006
Earlier work this paper cites.
Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation
Jamie Shotton, John Winn, Carsten Rother, and Antonio Criminisi · 2006
Earlier work this paper cites.
The Netflix prize
James Bennett, Stan Lanning, et al · 2007
Earlier work this paper cites.
Caltech-256 object category dataset, 2007
Gregory Griffin, Alex Holub, and Pietro Perona · 2007
Earlier work this paper cites.
Contemporary approaches to artificial general intelligence
Cassio Pennachin and Ben Goertzel · 2007
Earlier work this paper cites.
Essentials of general intelligence: The direct path to artificial general intelligence
Peter Voss · 2007
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei · 2009
Earlier work this paper cites.
The NIST 2008 metrics for machine translation challenge—overview, methodology, metrics, and results
Mark Przybocki, Kay Peterson, Sébastien Bronsart, and Gregory Sanders · 2009
Earlier work this paper cites.
The Pascal
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman · 2010
Earlier work this paper cites.
Fred Jelinek
Mark Liberman · 2010
Earlier work this paper cites.
Unbiased look at dataset bias
Antonio Torralba and Alexei A Efros · 2011
Earlier work this paper cites.
Is chess the drosophila of artificial intelligence? A social history of an algorithm
Nathan Ensmenger · 2012
Earlier work this paper cites.

Similar

Machine learning that matters
Kiri L. Wagstaff · 2012
Cited alongside, same era.
Enriching parallel corpora for statistical machine translation with semantic negation rephrasing
Dominikus Wetzel and Francis Bond · 2012
Cited alongside, same era.
Translating negation: A manual error analysis
Federico Fancellu and Bonnie Webber · 2015
Cited alongside, same era.
Comparison of image classification techniques using Caltech 101 dataset
Nur Shazwani Kamarudin, Mokhairi Makhtar, Syed Abdullah Fadzli, Mumtazimah Mohamad, Fatma Susilawati Mohamad, and Mohd Fadzil Abdul Kadir · 2015
Cited alongside, same era.
ImageNet Large Scale Visual Recognition Challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei · 2015
Cited alongside, same era.
Why and how Baidu cheated an artificial intelligence test
Tom Simonite · 2015
Cited alongside, same era.
What makes imagenet good for transfer learning?
Mi-Young Huh, Pulkit Agrawal, and A. Alexei Efros · 2016
Cited alongside, same era.
50 Years of Data Science
David Donoho · 2017
Cited alongside, same era.
UCI machine learning repository, 2017
Dheeru Dua and Casey Graff · 2017
Cited alongside, same era.
Towards linguistically generalizable NLP systems: A workshop and shared task
Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender · 2017
Cited alongside, same era.
Training computers to see internet pornography: Gender and sexual discrimination in computer vision science
Robert W Gehl, Lucas Moyer-Horner, and Sara K Yeo · 2017
Cited alongside, same era.
The data that transformed AI research—–and possibly the world
Dave Gershgorn · 2017
Cited alongside, same era.

Then

The limits of machine intelligence
Henry Shevlin, Karina Vold, Matthew Crosby, and Marta Halina · 2019
Later among the works it cites.
Energy and policy considerations for deep learning in NLP
Emma Strubell, Ananya Ganesh, and Andrew McCallum · 2019
Later among the works it cites.
GLUE: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman · 2019
Later among the works it cites.
Climbing towards NLU: On meaning, form, and understanding in the age of data
Emily M. Bender and Alexander Koller · 2020
Later among the works it cites.
Experience grounds language
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian · 2020
Later among the works it cites.
Value-laden disciplinary shifts in machine learning
Ravit Dotan and Smitha Milli · 2020
Later among the works it cites.
General purpose text embeddings from pre-trained language models for scalable inference
Jingfei Du, Myle Ott, Haoran Li, Xing Zhou, and Veselin Stoyanov · 2020
Later among the works it cites.
Utility is in the eye of the user: A critique of NLP leaderboard design
Kawin Ethayarajh and Dan Jurafsky · 2020
Later among the works it cites.
What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models
Allyson Ettinger · 2020
Later among the works it cites.
Race and gender
Timnit Gebru · 2020
Later among the works it cites.
Datasheets for datasets, 2020
Original
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford · 2020
Later among the works it cites.
Towards the systematic reporting of the energy and carbon footprints of machine learning
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau · 2020
Later among the works it cites.
It’s not a non-issue: Negation as a source of error in machine translation
Md Mosharaf Hossain, Antonios Anastasopoulos, Eduardo Blanco, and Alexis Palmer · 2020
Later among the works it cites.
Social biases in NLP models as barriers for persons with disabilities
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl · 2020
Later among the works it cites.
Lessons from archives: Strategies for collecting sociocultural data in machine learning
Eun Seo Jo and Timnit Gebru · 2020
Later among the works it cites.
Deep learning for generic object detection: A survey
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen · 2020
Later among the works it cites.
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Original
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M Bender, Emily Denton, and Alex Hanna · 2020
Later among the works it cites.
Pre-trained models for natural language processing: A survey
XiPeng Qiu, TianXiang Sun, YiGe Xu, YiGe Shao, Ning Dai, and XuanJing Huang · 2020
Later among the works it cites.
Beyond accuracy: Behavioral testing of NLP models with CheckList
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh · 2020
Later among the works it cites.
Green AI
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni · 2020
Later among the works it cites.
Disembodied machine learning: On the illusion of objectivity in NLP, 2020
Zeerak Waseem, Smarika Lulz, Joachim Bingel, and Isabelle Augenstein · 2020
Later among the works it cites.
On the dangers of stochastic parrots: Can language models be too big?
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell · 2021
Closest in time.
Large image datasets: A pyrrhic win for computer vision?
Abeba Birhane and Vinay Uday Prabhu · 2021
Closest in time.
On the opportunities and risks of foundation models
Original
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021
Closest in time.
What will it take to fix benchmarking in natural language understanding?
Samuel R. Bowman and George Dahl · 2021
Closest in time.
On the genealogy of machine learning datasets: A critical history of imagenet
Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, and Hilary Nicole · 2021
Closest in time.
Microsoft deberta surpasses human performance on the superglue benchmark, 2021
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen · 2021
Closest in time.
Measurement and fairness
Abigail Z. Jacobs and Hanna Wallach · 2021
Closest in time.
Why AI is harder than we think
Original
Melanie Mitchell · 2021
Closest in time.
“everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo · 2021
Closest in time.
Do datasets have politics? Disciplinary values in computer vision dataset development
Morgan Klaus Scheuerman, Alex Hanna, and Emily Denton · 2021
Closest in time.
Targeting the benchmark: On methodology in current natural language processing research
David Schlangen · 2021
Closest in time.
Underreporting of errors in NLG output, and what to do about it
Emiel van Miltenburg, Miruna Clinciu, Ondřej Dušek, Dimitra Gkatzia, Stephanie Inglis, Leo Leppänen, Saad Mahamood, Emma Manning, Stephanie Schoch, Craig Thomson, and Luou Wen · 2021
Closest in time.
TuringAdvice: A generative and dynamic evaluation of language use
Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…