2019

Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels

Chen, Pengfei, Liao, Benben, Chen, Guangyong et al.

Understand

Noisy labels are ubiquitous in real-world datasets, which poses a challenge for robustly training deep neural networks (DNNs) as DNNs usually have the high capacity to memorize the noisy labels.

In this paper, we find that the test accuracy can be quantitatively characterized in terms of the noise ratio in datasets.
In particular, the test accuracy is a quadratic function of the noise ratio in the case of symmetric noise, which explains the experimental findings previously published.
Based on our analysis, we apply cross-validation to randomly split noisy datasets, which identifies most samples that have correct labels.

Built on

A theory of the learnable
Valiant, L. G · 1984
Earlier work this paper cites.
Combining labeled and unlabeled data with co-training
Blum, A. and Mitchell, T · 1998
Earlier work this paper cites.
Adaptive and learning systems for signal processing communications, and control
Vapnik, V. N · 1998
Earlier work this paper cites.
Rademacher and gaussian complexities: Risk bounds and structural results
Bartlett, P. L. and Mendelson, S · 2002
Earlier work this paper cites.
Stability and generalization
Bousquet, O. and Elisseeff, A · 2002
Earlier work this paper cites.
Statistical learning: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization
Mukherjee, S., Niyogi, P., Poggio, T., and Rifkin, R · 2002
Earlier work this paper cites.
General conditions for predictivity in learning theory
Poggio, T., Rifkin, R., Mukherjee, S., and Niyogi, P · 2004
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation
Powers, D. M · 2011
Earlier work this paper cites.

Similar

Harvesting image databases from the web
Schroff, F., Criminisi, A., and Zisserman, A · 2011
Cited alongside, same era.
Learning everything about anything: Webly-supervised visual concept learning
Divvala, S. K., Farhadi, A., and Guestrin, C · 2014
Cited alongside, same era.
Learning from multiple annotators with varying expertise
Yan, Y., Rosales, R., Fung, G., Subramanian, R., and Dy, J · 2014
Cited alongside, same era.
A method for stochastic optimization
Kinga, D. and Adam, J. B · 2015
Cited alongside, same era.
Training deep neural networks on noisy labels with bootstrapping
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A · 2015
Cited alongside, same era.
The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality
Amsaleg, L., Bailey, J., Barbe, D., Erfani, S., Houle, M. E., Nguyen, V., and Radovanović, M · 2017
Cited alongside, same era.
A closer look at memorization in deep networks
Arpit, D., Jastrzębski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al · 2017
Cited alongside, same era.
Learning to aggregate ordinal labels by maximizing separating width
Chen, G., Zhang, S., Lin, D., Huang, H., and Heng, P. A · 2017
Cited alongside, same era.
Training deep neural-networks using a noise adaptation layer
Goldberger, J. and Ben-Reuven, E · 2017
Cited alongside, same era.
Webvision database: Visual learning and understanding from web data
Original
Li, W., Wang, L., Li, W., Agustsson, E., and Van Gool, L · 2017
Cited alongside, same era.
Deep residual learning for image recognition
He, K., Zhang, X., Ren, S., and Sun, J
Cited in the paper.
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., and Sun, J
Cited in the paper.

Then

Decoupling" when to update" from" how to update"
Malach, E. and Shalev-Shwartz, S · 2017
Later among the works it cites.
Making deep neural networks robust to label noise: A loss correction approach
Patrini, G., Rozza, A., Menon, A. K., Nock, R., and Qu, L · 2017
Later among the works it cites.
Inception-v4, inception-resnet and the impact of residual connections on learning
Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A · 2017
Later among the works it cites.
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O · 2017
Later among the works it cites.
Co-teaching: robust training deep neural networks with extremely noisy labels
Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., and Sugiyama, M · 2018
Later among the works it cites.
Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels
Jiang, L., Zhou, Z., Leung, T., Li, L.-J., and Fei-Fei, L · 2018
Later among the works it cites.
Dimensionality-driven learning with noisy labels
Ma, X., Wang, Y., Houle, M. E., Zhou, S., Erfani, S. M., Xia, S.-T., Wijewickrema, S., and Bailey, J · 2018
Later among the works it cites.
Learning to reweight examples for robust deep learning
Ren, M., Zeng, W., Yang, B., and Urtasun, R · 2018
Later among the works it cites.
Joint optimization framework for learning with noisy labels
Tanaka, D., Ikami, D., Yamasaki, T., and Aizawa, K · 2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…