2020

Characterizing Structural Regularities of Labeled Data in Overparameterized Models

Jiang, Ziheng, Zhang, Chiyuan, Talwar, Kunal et al.

Understand

Humans are accustomed to environments that contain both regularities and exceptions.

For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance.
Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms.
We analyze how individual instances are treated by a model via a consistency score.

Built on

On Learning the Past Tenses of English Verbs , pp. 216–271
Rumelhart, D. E. and McClelland, J. L · 1986
Earlier work this paper cites.
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P · 1998
Earlier work this paper cites.
Lof: identifying density-based local outliers
Breunig, M. M., Kriegel, H.-P., Ng, R. T., and Sander, J · 2000
Earlier work this paper cites.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A · 2009
Earlier work this paper cites.
Reading digits in natural images with unsupervised feature learning
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y · 2011
Earlier work this paper cites.
TensorFlow: Large-scale machine learning on heterogeneous systems, 2015
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X · 2015
Earlier work this paper cites.
Adam: A method for stochastic optimization
Kingma, D. P. and Ba, J · 2015
Earlier work this paper cites.
Imagenet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al · 2015
Earlier work this paper cites.
Train faster, generalize better: Stability of stochastic gradient descent
Hardt, M., Recht, B., and Singer, Y · 2016
Earlier work this paper cites.
Improving generalization performance by switching from adam to sgd
Original
Keskar, N. S. and Socher, R · 2017
Earlier work this paper cites.

Similar

The marginal value of adaptive gradient methods in machine learning
Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., and Recht, B · 2017
Cited alongside, same era.
Prototypical examples in deep learning: Metrics, characteristics, and utility
Carlini, N., Erlingsson, U., and Papernot, N · 2018
Cited alongside, same era.
Neural tangent kernel: Convergence and generalization in neural networks
Jacot, A., Gabriel, F., and Hongler, C · 2018
Cited alongside, same era.
Adapted deep embeddings: A synthesis of methods for k-shot inductive transfer learning
Scott, T., Ridgeway, K., and Mozer, M. C · 2018
Cited alongside, same era.
Blockdrop: Dynamic inference paths in residual networks
Wu, Z., Nagarajan, T., Kumar, A., Rennie, S., Davis, L. S., Grauman, K., and Feris, R · 2018
Cited alongside, same era.
Adaptive gradient methods with dynamic bound of learning rate
Luo, L., Xiong, Y., Liu, Y., and Sun, X · 2019
Cited alongside, same era.
Do deep neural networks learn shallow learnable examples first?
Mangalam, K. and Prabhu, V. U · 2019
Cited alongside, same era.
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I · 2019
Cited alongside, same era.
Data parameters: A new family of parameters for learning a differentiable curriculum
Saxena, S., Tuzel, O., and DeCoste, D · 2019
Cited alongside, same era.
Efficientnet: Rethinking model scaling for convolutional neural networks
Original
Tan, M. and Le, Q. V · 2019
Cited alongside, same era.
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G
Cited in the paper.
Big Self-Supervised models are strong Semi-Supervised learners
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G
Cited in the paper.

Then

An empirical study of example forgetting during deep neural network learning
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J · 2019
Later among the works it cites.
Does learning require memorization? A short tale about a long tail
Feldman, V · 2020
Closest in time.
What neural networks memorize and why: Discovering the long tail via influence estimation
Feldman, V. and Zhang, C · 2020
Closest in time.
Bootstrap your own latent: A new approach to Self-Supervised learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., Piot, B., Kavukcuoglu, K., Munos, R., and Valko, M · 2020
Closest in time.
Detecting noisy training data with loss curves
Pleiss, G., Zhang, T., Elenberg, E. R., and Weinberger, K. Q · 2020
Closest in time.
Emerging properties in self-supervised vision transformers
Original
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A · 2021
Closest in time.
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N · 2021
Closest in time.
Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet
Original
Melas-Kyriazi, L · 2021
Closest in time.
Mlp-mixer: An all-mlp architecture for vision
Original
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Keysers, D., Uszkoreit, J., Lucic, M., et al · 2021
Closest in time.
Resmlp: Feedforward networks for image classification with data-efficient training
Original
Touvron, H., Bojanowski, P., Caron, M., Cord, M., El-Nouby, A., Grave, E., Joulin, A., Synnaeve, G., Verbeek, J., and Jégou, H · 2021
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…