2024

The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?

Besiroglu, Tamay, Bergerson, Sage Andrus, Michael, Amelia et al.

Understand

There are pronounced differences in the extent to which industrial and academic AI labs use computing resources.

We provide a data-driven survey of the role of the compute divide in shaping machine learning research.
We show that a compute divide has coincided with a reduced representation of academic-only research teams in compute intensive research topics, especially foundation models.
We argue that, academia will likely play a smaller role in advancing the associated techniques, providing critical evaluation and scrutiny, and in the diffusion of such models.

Built on

“Language models are few-shot learners”
Tom Brown et al · 1901
Earlier work this paper cites.
“Published papers, tacit competencies and corporate management of the public/private character of knowledge”
Diana Hicks · 1995
Earlier work this paper cites.
“Do pharmaceutical sales respond to scientific evidence?”
Pierre Azoulay · 2002
Earlier work this paper cites.
“Defensive use of publications in an intellectual property strategy”
Bill Barrett · 2002
Earlier work this paper cites.
“R&D cooperation and spillovers: some empirical evidence from Belgium”
Bruno Cassiman and Reinhilde Veugelers · 2002
Earlier work this paper cites.
“Some simple economics of open source”
Josh Lerner and Jean Tirole · 2002
Earlier work this paper cites.
“Open source software from commercial firms–tools, complements, and collective invention”
Joachim Henkel · 2004
Earlier work this paper cites.
“The ethics of science: An introduction”
David Resnik · 2005
Earlier work this paper cites.
“High performance convolutional neural networks for document processing”
Kumar Chellapilla, Sidd Puri and Patrice Simard · 2006
Earlier work this paper cites.
“A man on the inside: Unlocking communities as complementary assets”
Linus Dahlander and Martin Wallin · 2006
Earlier work this paper cites.
“Outsourcing to an unknown workforce: Exploring opensurcing as a global sourcing strategy”
Pär Ågerfalk and Brian Fitzgerald · 2008
Earlier work this paper cites.
“The business of open source”
Richard Watson et al · 2008
Earlier work this paper cites.
“The first-mover advantage in scientific publication”
Mark Newman · 2009
Earlier work this paper cites.
“Large-scale object recognition with CUDA-accelerated hierarchical neural networks”
Rafael Uetz and Sven Behnke · 2009
Earlier work this paper cites.
“Handwritten digit recognition with a committee of deep neural nets on gpus”
Original
Dan Cireşan, Ueli Meier, Luca Gambardella and Jürgen Schmidhuber · 2011
Earlier work this paper cites.
“Getting competition down to a science: The effects of technological competition on firms’ scientific publications”
Francisco Polidoro and Matt Theeke · 2012
Earlier work this paper cites.
“Cui bono? The selective revealing of knowledge and its implications for innovative activity”
Oliver Alexy, Gerard George and Ammon Salter · 2013
Earlier work this paper cites.
“Activity traces and signals in software developer recruitment and hiring”
Jennifer Marlow and Laura Dabbish · 2013
Earlier work this paper cites.
“1.1. 1 Industry and Academia: Why Practitioners and Researchers are Disconnected”
Gerrit Muller · 2014
Earlier work this paper cites.
“Deep learning”
Yann LeCun, Yoshua Bengio and Geoffrey Hinton · 2015
Earlier work this paper cites.
“Deep learning in neural networks: An overview”
Jürgen Schmidhuber · 2015
Earlier work this paper cites.
“The motivations, institutions and organization of university-industry collaborations in the Netherlands”
Isabel Bodas and Bart Verspagen · 2017
Earlier work this paper cites.
“A manifesto for reproducible science”
Marcus Munafò et al · 2017
Earlier work this paper cites.
“Bert: Pre-training of deep bidirectional transformers for language understanding”
Original
Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova · 2018
Earlier work this paper cites.
“Collaborate or consolidate: Assessing the competitive effects of production joint ventures”
Nicolas Aguelakakis and Aleksandr Yankelevich · 2019
Earlier work this paper cites.

Similar

“The role of cooperation in responsible AI development”
Original
Amanda Askell, Miles Brundage and Gillian Hadfield · 2019
Cited alongside, same era.
“Artificial intelligence for social good”
Original
Gregory Hager et al · 2019
Cited alongside, same era.
“Gpipe: Efficient training of giant neural networks using pipeline parallelism”
Yanping Huang et al · 2019
Cited alongside, same era.
“Are sixteen heads really better than one?”
Paul Michel, Omer Levy and Graham Neubig · 2019
Cited alongside, same era.
“Megatron-lm: Training multi-billion parameter language models using model parallelism”
Original
Mohammad Shoeybi et al · 2019
Cited alongside, same era.
“Release strategies and the social impacts of language models”
Original
Irene Solaiman et al · 2019
Cited alongside, same era.
“The tension between openness and prudence in responsible AI research”, 2019
Jess Whittlestone and Aviv Ovadya · 2019
Cited alongside, same era.
“The de-democratization of AI: Deep learning and the compute divide in artificial intelligence research”
Original
Nur Ahmed and Muntasir Wahed · 2020
Cited alongside, same era.
“Toward trustworthy AI development: mechanisms for supporting verifiable claims”
Original
Miles Brundage et al · 2020
Cited alongside, same era.
“ProtTrans: towards cracking the language of Life’s code through self-supervised deep learning and high performance computing”
Original
Ahmed Elnaggar et al · 2020
Cited alongside, same era.
“Open Source Research Software”
Wilhelm Hasselbring et al · 2020
Cited alongside, same era.
“A narrowing of AI research?”
Original
Joel Klinger, Juan Mateos-Garcia and Konstantinos Stathoulopoulos · 2020
Cited alongside, same era.

Then

“Luna: Linear unified nested attention”
Xuezhe Ma et al · 2021
Later among the works it cites.
“Can fMRI reveal the representation of syntactic structure in the brain?”
Aniketh Reddy and Leila Wehbe · 2021
Later among the works it cites.
“Training neural networks with fixed sparse masks”
Yi-Lin Sung, Varun Nair and Colin Raffel · 2021
Later among the works it cites.
“Scientific Labor Market and Firm-level Appropriation Strategy in Artificial Intelligence Research”
Nur Ahmed · 2022
Later among the works it cites.
“Compute Funds and Pre-trained Models” Research Posts, April 11, 2022
Markus Anderljung, Lennart Heim and Toby Shevlane · 2022
Later among the works it cites.
“Varuna: scalable, low-cost training of massive deep learning models”
Sanjith Athlur et al · 2022
Later among the works it cites.
“Defining the role of open source software in research reproducibility”
Original
Lorena Barba · 2022
Later among the works it cites.
“Compute and Antitrust”
Haydn Belfield and Shin-Shin Hua · 2022
Later among the works it cites.
“BigScience Language Open-science Open-access Multilingual (BLOOM) Language Model” All collaborators are either volunteers or have an agreement with their employer
BigScience · 2022
Later among the works it cites.
“Strategic Real Option Exercising and Second-mover Advantage”, 2022
Min Dai, Zhaoli Jiang and Neng Wang · 2022
Later among the works it cites.
“Independent Review of the Future of Compute” An independent report on the UK compute needs over the next decade, Online, 2022
Department for Science, Innovation and Technology · 2022
Later among the works it cites.
“Recent Trends in China’s Large-Scale Pre-Trained AI Models Landscape”, 2022
Jeffrey Ding and Jenny. Xiao · 2022
Later among the works it cites.
“Predictability and surprise in large generative models”
Deep Ganguli et al · 2022
Later among the works it cites.
“AI, Diversity, and Collaboration” Last accessed: 7 Feb 2023, 2022
Juan-Mateos Garcia · 2022
Later among the works it cites.
“How Open Source Machine Learning Software Shapes AI”
Max Langenkamp and Daniel Yue · 2022
Later among the works it cites.
“The Time Is Now to Develop Community Norms for the Release of Foundation Models”, 2022
Percy Liang, Rishi Bommasani, Kathleen. Creel and Rob Reich · 2022
Later among the works it cites.
“Aligning Language Models To Follow Instructions”, https://openai.com/research/instruction-following , 2022
Ryan Lowe and Jan Leike · 2022
Later among the works it cites.
“Compute trends across three eras of machine learning”
Original
Jaime Sevilla et al · 2022
Later among the works it cites.
“Structured Access: An Emerging Paradigm for Safe AI Deployment”, 2022
Toby Shevlane · 2022
Later among the works it cites.
“You reap what you sow: On the challenges of bias evaluation under multilingual settings”
Zeerak Talat et al · 2022
Later among the works it cites.
“When big AI labs refuse to open source their models, the community steps in”, 2022
Kyle Wiggers · 2022
Later among the works it cites.
“Opt: Open pre-trained transformer language models”
Original
Susan Zhang et al · 2022
Later among the works it cites.
“The growing influence of industry in AI research”
Nur Ahmed, Muntasir Wahed and Neil Thompson · 2023
Later among the works it cites.
“Strengthening and Democratizing the U.S. Artificial Intelligence Innovation Ecosystem: An Implementation Plan for a National Artificial Intelligence Research Resource”, 2023
National Force · 2023
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…