2020

Identifying Mislabeled Data using the Area Under the Margin Ranking

Pleiss, Geoff, Zhang, Tianyi, Elenberg, Ethan R. et al.

Understand

Not all data in a typical training set help with generalization; some samples can be overly ambiguous or outrightly mislabeled.

This paper introduces a new method to identify such samples and mitigate their impact when training neural networks.
At the heart of our algorithm is the Area Under the Margin (AUM) statistic, which exploits differences in the training dynamics of clean and mislabeled samples.
A simple procedure - adding an extra class populated with purposefully mislabeled threshold samples - learns a AUM upper bound that isolates mislabeled data.

Built on

The nature of statistical learning theory
V. Vapnik · 1995
Earlier work this paper cites.
For valid generalization the size of the weights is more important than the size of the network
P. L. Bartlett · 1997
Earlier work this paper cites.
Boosting the margin: A new explanation for the effectiveness of voting methods
R. E. Shapire, Y. Freund, P. Bartlett, and W. S. Lee · 1998
Earlier work this paper cites.
Identifying mislabeled training data
C. E. Brodley and M. A. Friedl · 1999
Earlier work this paper cites.
Experiments with noise filtering in a medical domain
D. Gamberger, R. Boskovic, N. Lavrac, and C. Groselj · 1999
Earlier work this paper cites.
Eliminating class noise in large datasets
X. Zhu, X. Wu, and Q. Chen · 2003
Earlier work this paper cites.
The mnist database of handwritten digits
Y. LeCun and C. Cortes · 2005
Earlier work this paper cites.
Curriculum learning
Y. Bengio, J. Louradour, R. Collobert, and J. Weston · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
A. Krizhevsky et al · 2009
Earlier work this paper cites.
Distance metric learning for large margin nearest neighbor classification
K. Q. Weinberger and L. K. Saul · 2009
Earlier work this paper cites.
Learning with noisy labels
N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari · 2013
Earlier work this paper cites.
Classification in the presence of label noise: A survey
B. Frenay and M. Verleysen · 2014
Earlier work this paper cites.
Training deep neural networks on noisy labels with bootstrapping
Original
S. E. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich · 2014
Earlier work this paper cites.
Training convolutional networks with noisy labels
S. Sukhbaatar, J. Bruna, M. Paluri, L. D. Bourdev, and R. Fergus · 2015
Earlier work this paper cites.
Learning from massive noisy labeled data for image classification
T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang · 2015
Earlier work this paper cites.
Deep residual learning for image recognition
K. He, X. Zhang, S. Ren, and J. Sun · 2016
Earlier work this paper cites.
Learning visual features from large weakly supervised data
A. Joulin, L. van der Maaten, A. Jabri, and N. Vasilache · 2016
Earlier work this paper cites.
A closer look at memorization in deep networks
D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, and S. Lacoste-Julien · 2017
Earlier work this paper cites.
Spectrally-normalized margin bounds for neural networks
P. L. Bartlett, D. J. Foster, and M. J. Telgarsky · 2017
Earlier work this paper cites.
Training deep neural-networks using a noise adaptation layer
J. Goldberger and E. Ben-Reuven · 2017
Earlier work this paper cites.
Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels
L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei · 2017
Earlier work this paper cites.

Similar

On large-batch training for deep learning: Generalization gap and sharp minima
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang · 2017
Cited alongside, same era.
Learning with confident examples: Rank pruning for robust classification with noisy labels
C. G. Northcutt, T. Wu, and I. L. Chuang · 2017
Cited alongside, same era.
Understanding deep learning requires rethinking generalization
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals · 2017
Cited alongside, same era.
Stronger generalization bounds for deep nets via a compression approach
S. Arora, R. Ge, B. Neyshabur, and Y. Zhang · 2018
Cited alongside, same era.
Large margin deep networks for classification
G. Elsayed, D. Krishnan, H. Mobahi, K. Regan, and S. Bengio · 2018
Cited alongside, same era.
Co-teaching: Robust training of deep neural networks with extremely noisy labels
B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. W.-H. Tsang, and M. Sugiyama · 2018
Cited alongside, same era.
Using trusted data to train deep networks on labels corrupted by severe noise
D. Hendrycks, M. Mazeika, D. Wilson, and K. Gimpel · 2018
Cited alongside, same era.
Smooth neighbors on teacher graphs for semi-supervised learning
Y. Luo, J. Zhu, M. Li, Y. Ren, and B. Zhang · 2018
Cited alongside, same era.
Dimensionality-driven learning with noisy labels
X. Ma, Y. Wang, M. E. Houle, S. Zhou, S. M. Erfani, S.-T. Xia, S. N. R. Wijewickrema, and J. Bailey · 2018
Cited alongside, same era.
Learning to reweight examples for robust deep learning
M. Ren, W. Zeng, B. Yang, and R. Urtasun · 2018
Cited alongside, same era.
Data dropout: Optimizing training data for convolutional neural networks
T. Wang, J. Huan, and B. Li · 2018
Cited alongside, same era.
mixup: Beyond empirical risk minimization
H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz · 2018
Cited alongside, same era.

Then

Predicting the generalization gap in deep networks with margin distributions
Y. Jiang, D. Krishnan, H. Mobahi, and S. Bengio · 2019
Later among the works it cites.
Learning to learn from noisy labeled data
J. Li, Y. Wong, Q. Zhao, and M. S. Kankanhalli · 2019
Later among the works it cites.
Fisher-rao metric, geometry, and complexity of neural networks
T. Liang, T. Poggio, A. Rakhlin, and J. Stokes · 2019
Later among the works it cites.
Towards understanding the role of over-parametrization in generalization of neural networks
B. Neyshabur, Z. Li, S. Bhojanapalli, Y. LeCun, and N. Srebro · 2019
Later among the works it cites.
Confident learning: Estimating uncertainty in dataset labels
Original
C. G. Northcutt, L. Jiang, and I. L. Chuang · 2019
Later among the works it cites.
Pytorch: An imperative style, high-performance deep learning library
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al · 2019
Later among the works it cites.
Do imagenet classifiers generalize to imagenet
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar · 2019
Later among the works it cites.
Data parameters: A new family of parameters for learning a differentiable curriculum
S. Saxena, O. Tuzel, and D. DeCoste · 2019
Later among the works it cites.
Learning with bad training data via iterative trimmed loss minimization
Y. Shen and S. Sanghavi · 2019
Later among the works it cites.
L DMI {L_{\text{DMI}}} : An information-theoretic noise-robust loss function
Y. Xu, P. Cao, Y. Kong, and Y. Wang · 2019
Later among the works it cites.
Probabilistic end-to-end noise correction for learning with noisy labels
K. Yi and J. Wu · 2019
Later among the works it cites.
How does disagreement help generalization against label corruption?
X. Yu, B. Han, J. Yao, G. Niu, I. W. Tsang, and M. Sugiyama · 2019
Later among the works it cites.
Do we train on test data? purging CIFAR of near-duplicates
B. Barz and J. Denzler · 2020
Closest in time.
Fantastic generalization measures and where to find them
Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio · 2020
Closest in time.
Peer loss functions: Learning from noisy labels without knowing noise rates
Y. Liu and H. Guo · 2020
Closest in time.
Deep double descent: Where bigger models and more data hurt
P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever · 2020
Closest in time.
Self: Learning to filter noisy labels with self-ensembling
D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox · 2020
Closest in time.
Towards robust learning with different label noise distributions
D. Ortego, E. Arazo, P. Albert, N. E. O’Connor, and K. McGuinness · 2020
Closest in time.
Dataset cartography: Mapping and diagnosing datasets with training dynamics
S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi · 2020
Closest in time.
Searching to exploit memorization effect in learning from corrupted labels
Q. Yao, H. Yang, B. Han, G. Niu, and J. Kwok · 2020
Closest in time.
Distilling effective supervision from severe label noise
Z. Zhang, H. Zhang, S. O. Arik, H. Lee, and T. Pfister · 2020
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…