Fetching the paper…
Reading the bibliography…
Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning.
Spin glass theory and beyond
M Mezard, G Parisi, and M A Virasoro · 1987
Earlier work this paper cites.
The space of interactions in neural network models
E Gardner · 1988
Earlier work this paper cites.
Statistical mechanics of learning from examples
H S Seung, H Sompolinsky, and N Tishby · 1992
Earlier work this paper cites.
Information, prediction, and query by committee
Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby · 1992
Earlier work this paper cites.
Statistical Mechanics of Learning
A Engel and C V den Broeck · 2001
Earlier work this paper cites.
Fast kernel classifiers with online and active learning
Antoine Bordes, Seyda Ertekin, Jason Weston, Léon Botton, and Nello Cristianini · 2005
Earlier work this paper cites.
Active learning literature survey
Burr Settles · 2009
Earlier work this paper cites.
Statistical mechanics of complex neural systems and high dimensional data
Madhu Advani, Subhaneil Lahiri, and Surya Ganguli · 2013
Earlier work this paper cites.
ImageNet large scale visual recognition challenge
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al · 2015
Earlier work this paper cites.
Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Earlier work this paper cites.
Statistical physics of inference: thresholds and algorithms
Lenka Zdeborová and Florent Krzakala · 2016
Earlier work this paper cites.
Deep learning scaling is predictable, empirically
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou · 2017
Earlier work this paper cites.
Active learning for convolutional neural networks: A core-set approach
Ozan Sener and Silvio Savarese · 2017
Earlier work this paper cites.
Methods and measurements to compare men against machines
Felix A Wichmann, David HJ Janssen, Robert Geirhos, Guillermo Aguilar, Heiko H Schütt, Marianne Maertens, and Matthias Bethge · 2017
Earlier work this paper cites.
Exploring the limits of weakly supervised pretraining
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten · 2018
Earlier work this paper cites.
Generalisation in humans and deep neural networks
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann · 2018
Earlier work this paper cites.
An empirical study of example forgetting during deep neural network learning
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon · 2019
Earlier work this paper cites.
Semantic redundancies in Image-Classification datasets: The 10% you don’t need
V Birodkar, H Mobahi, and S Bengio · 2019
Earlier work this paper cites.
Active online learning in the binary perceptron problem
Hai-Jun Zhou · 2019
Cited alongside, same era.
Pytorch image models
Ross Wightman · 2019
Cited alongside, same era.
Survey on deep learning with class imbalance
Justin M Johnson and Taghi M Khoshgoftaar · 2019
Cited alongside, same era.
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel · 2019
Cited alongside, same era.
Learning robust global representations by penalizing local predictive power
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing · 2019
Cited alongside, same era.
Scaling laws for neural language models
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei · 2020
Training data subset search with ensemble active learning
Kashyap Chitta, José M Álvarez, Elmar Haussmann, and Clément Farabet · 2021
Later among the works it cites.
On the opportunities and risks of foundation models
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al · 2021
Later among the works it cites.
Estimating informativeness of samples with smooth unique information
Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto · 2021
Later among the works it cites.
Active learning at the ImageNet scale
Zeyad Ali Sami Emam, Hong-Min Chu, Ping-Yeh Chiang, Wojciech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein · 2021
Later among the works it cites.
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Cited alongside, same era.
Scaling laws for autoregressive generative modeling
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al · 2020
Cited alongside, same era.
A constructive prediction of the generalization error across scales
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit · 2020
Cited alongside, same era.
What neural networks memorize and why: Discovering the long tail via influence estimation
Vitaly Feldman and Chiyuan Zhang · 2020
Cited alongside, same era.
Coresets for data-efficient training of machine learning models
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec · 2020
Cited alongside, same era.
Statistical mechanics of deep learning
Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli · 2020
Cited alongside, same era.
Unsupervised learning of visual features by contrasting cluster assignments
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin · 2020
Cited alongside, same era.
Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher D Manning · 2021
Later among the works it cites.
Explaining neural scaling laws
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma · 2021
Later among the works it cites.
Scaling Laws for Deep Learning
Jonathan S. Rosenfeld · 2021
Later among the works it cites.
Large deviations in the perceptron model and consequences for active learning, 2021
Hugo Cui, Luca Saglietti, and Lenka Zdeborovà · 2021
Later among the works it cites.
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al · 2021
Later among the works it cites.
PASS: An ImageNet replacement for self-supervised pretraining without humans
Yuki M Asano, Christian Rupprecht, Andrew Zisserman, and Andrea Vedaldi · 2021
Later among the works it cites.
Partial success in closing the gap between human and machine vision
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel · 2021
Later among the works it cites.
Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt · 2021
Later among the works it cites.
Training compute-optimal large language models
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al · 2022
Closest in time.
Trivial or impossible—dichotomous data difficulty masks model differences (on ImageNet and beyond)
Kristof Meding, Luca M. Schulze Buschoff, Robert Geirhos, and Felix A. Wichmann · 2022
Closest in time.
Scaling laws from the data manifold dimension
Utkarsh Sharma and Jared Kaplan · 2022
Closest in time.
Hierarchical text-conditional image generation with clip latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen · 2022
Closest in time.
Palm: Scaling language modeling with pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al · 2022
Closest in time.