Fetching the paper…

Reading the bibliography…

2020

Robust and On-the-fly Dataset Denoising for Image Classification

Song, Jiaming, Hu, Lunjia, Auli, Michael et al.

Original
PDF
Post on X

Understand

Memorization in over-parameterized neural networks could severely hurt generalization in the presence of mislabeled examples.

However, mislabeled examples are hard to avoid in extremely large datasets collected with weak supervision.
We address this problem by reasoning counterfactually about the loss distribution of examples with uniform random labels had they were trained with the real examples, and use this information to remove noisy examples from the training set.
First, we observe that examples with uniform random labels have higher losses when trained with stochastic gradient descent under large learning rates.

Robust and On-the-fly Dataset Denoising for Image Classification · Around

Built on

Hochreiter, S., Schmidhuber, J.: SIMPLIFYING NEURAL NETS BY DISCOVERING FLAT MINIMA. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems 7, pp. 529–536. MIT Press (1995)
1995
Earlier work this paper cites.
Brodley, C.E., Friedl, M.A., Others: Identifying and eliminating mislabeled training instances. In: Proceedings of the National Conference on Artificial Intelligence. pp. 799–805 (1996)
1996
Earlier work this paper cites.
Cretu, G.F., Stavrou, A., Locasto, M.E., Stolfo, S.J., Keromytis, A.D.: Casting out demons: Sanitizing training data for anomaly sensors. In: Security and Privacy, 2008. SP 2008. IEEE Symposium on. pp. 81–95. IEEE (2008)
2008
Earlier work this paper cites.
Liu, T., Tao, D.: Classification with noisy labels by importance reweighting. arXiv preprint arXiv:1411.7718 (November 2014)
Original
2014
Earlier work this paper cites.
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., Rabinovich, A.: Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596 (December 2014)
Original
2014
Earlier work this paper cites.
Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., Fergus, R.: Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080 (June 2014)
Original
2014
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385 (December 2015)
Original
2015
Earlier work this paper cites.
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing Human-Level performance on ImageNet classification. arXiv preprint arXiv:1502.01852 (February 2015)
Original
2015
Earlier work this paper cites.
Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)
Original
2015
Earlier work this paper cites.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. International journal of computer vision 115
2015
Earlier work this paper cites.
Xiao, T., Xia, T., Yang, Y., Huang, C., Wang, X.: Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2691–2699 (2015)
2015
Earlier work this paper cites.
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., Zecchina, R.: Entropy-SGD: Biasing gradient descent into wide valleys. arXiv preprint arXiv:1611.01838 (November 2016)
Original
2016
Earlier work this paper cites.
Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On Large-Batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 (September 2016)
Original
2016
Earlier work this paper cites.
Krishna, R.A., Hata, K., Chen, S., Kravitz, J., Shamma, D.A., Fei-Fei, L., Bernstein, M.S.: Embracing error to enable rapid crowdsourcing. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. pp. 3167–3179. CHI ’16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2858036.2858115
2016
Earlier work this paper cites.
Loshchilov, I., Hutter, F.: SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (August 2016)
Original
2016
Earlier work this paper cites.
Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 761–769 (2016)
2016
Earlier work this paper cites.

Similar

Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, Inception-ResNet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261 (February 2016)
Original
2016
Cited alongside, same era.
Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (May 2016)
Original
2016
Cited alongside, same era.
Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 (November 2016)
Original
2016
Cited alongside, same era.
Arpit, D., Jastrzkebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., Lacoste-Julien, S.: A closer look at memorization in deep networks. arXiv preprint arXiv:1706.05394 (June 2017)
Original
2017
Cited alongside, same era.

Then

Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., Belongie, S.J.: Learning from noisy Large-Scale datasets with minimal supervision. In: CVPR. pp. 6575–6583 (2017)
2017
Later among the works it cites.
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization (October 2017)
2017
Later among the works it cites.
Chaudhari, P., Soatto, S.: Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In: 2018 Information Theory and Applications Workshop (ITA). pp. 1–10. IEEE (2018)
2018
Later among the works it cites.
Du, S.S., Hu, W., Lee, J.D.: Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. arXiv preprint arXiv:1806.00900 (June 2018)
Original
2018
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Ghosh, A., Kumar, H., Sastry, P.S.: Robust loss functions under label noise for deep neural networks. In: AAAI. pp. 1919–1925 (2017)

2017

Cited alongside, same era.

Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677 (June 2017)

2017

Cited alongside, same era.

Hoffer, E., Hubara, I., Soudry, D.: Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30, pp. 1731–1741. Curran Associates, Inc. (2017)

2017

Cited alongside, same era.

Jiang, L., Zhou, Z., Leung, T., Li, L.J., Fei-Fei, L.: MentorNet: Learning Data-Driven curriculum for very deep neural networks on corrupted labels. arXiv preprint arXiv:1712.05055 (December 2017)

2017

Cited alongside, same era.

Li, W., Wang, L., Li, W., Agustsson, E., Van Gool, L.: WebVision database: Visual learning and understanding from web data. arXiv preprint arXiv:1708.02862 (August 2017)

2017

Cited alongside, same era.

Li, Y., Ma, T., Zhang, H.: Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. arXiv preprint arXiv:1712.09203 (December 2017)

2017

Cited alongside, same era.

Mandt, S., Hoffman, M.D., Blei, D.M.: Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research 18

2017

Cited alongside, same era.

Neyshabur, B.: Implicit regularization in deep learning. arXiv preprint arXiv:1709.01953 (September 2017)

2017

Cited alongside, same era.

Guo, S., Huang, W., Zhang, H., Zhuang, C., Dong, D., Scott, M.R., Huang, D.: Curriculumnet: Weakly supervised learning from large-scale web images. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 135–150 (2018)

2018

Later among the works it cites.

Hendrycks, D., Mazeika, M., Wilson, D., Gimpel, K.: Using trusted data to train deep networks on labels corrupted by severe noise. arXiv preprint arXiv:1802.05300 (February 2018)

2018

Later among the works it cites.

Kleinberg, R., Li, Y., Yuan, Y.: An alternative view: When does SGD escape local minima? arXiv preprint arXiv:1802.06175 (February 2018)

2018

Later among the works it cites.

Lee, K.H., He, X., Zhang, L., Yang, L.: Cleannet: Transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5447–5456 (2018)

2018

Later among the works it cites.

Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., van der Maaten, L.: Exploring the limits of weakly supervised pretraining. arXiv preprint arXiv:1805.00932 (May 2018)

Original

2018

Later among the works it cites.

Ren, M., Zeng, W., Yang, B., Urtasun, R.: Learning to reweight examples for robust deep learning. arXiv preprint arXiv:1803.09050 (March 2018)

2018

Later among the works it cites.

Tanaka, D., Ikami, D., Yamasaki, T., Aizawa, K.: Joint optimization framework for learning with noisy labels. arXiv preprint arXiv:1803.11364 (2018)

2018

Later among the works it cites.

Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. In: Advances in neural information processing systems. pp. 8778–8788 (2018)

2018

Later among the works it cites.

Luo, Y., Zhu, J., Pfister, T.: A simple yet effective baseline for robust deep learning with noisy labels. arXiv preprint arXiv:1909.09338 (2019)

2019

Later among the works it cites.

Xu, Y., Cao, P., Kong, Y., Wang, Y.: L_dmi: An information-theoretic noise-robust loss function. arXiv preprint arXiv:1909.03388 (2019)

2019

Later among the works it cites.

Yao, J., Wu, H., Zhang, Y., Tsang, I.W., Sun, J.: Safeguarded dynamic label regression for noisy supervision. AAAI (2019)

2019

Later among the works it cites.