2017

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

Jiang, Lu, Zhou, Zhengyuan, Leung, Thomas et al.

Understand

Recent deep networks are capable of memorizing the entire data even when the labels are completely random.

To overcome the overfitting on corrupted labels, we propose a novel technique of learning another neural network, called MentorNet, to supervise the training of the base deep networks, namely, StudentNet.
During training, MentorNet provides a curriculum (sample weighting scheme) for StudentNet to focus on the sample the label of which is probably correct.
Unlike the existing curriculum that is usually predefined by human experts, MentorNet learns a data-driven curriculum dynamically with StudentNet.

Built on

Robust estimation of a location parameter
Huber, P. J. et al · 1964
Earlier work this paper cites.
Robust estimation of a location parameter
Huber, P. J. et al · 1964
Earlier work this paper cites.
Information geometry and alternating minimization procedures
Csiszar, I · 1984
Earlier work this paper cites.
The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields
Black, M. J. and Anandan, P · 1996
Earlier work this paper cites.
Enhancing sparsity by reweighted l1 minimization
Candes, E. J., Wakin, M. B., and Boyd, S. P · 2008
Earlier work this paper cites.
Enhancing sparsity by reweighted l1 minimization
Candes, E. J., Wakin, M. B., and Boyd, S. P · 2008
Earlier work this paper cites.
Curriculum learning
Bengio, Y., Louradour, J., Collobert, R., and Weston, J · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
Imagenet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. and Hinton, G · 2009
Earlier work this paper cites.
Self-paced learning for latent variable models
Kumar, M. P., Packer, B., and Koller, D · 2010
Earlier work this paper cites.
Word representations: a simple and general method for semi-supervised learning
Turian, J., Ratinov, L., and Bengio, Y · 2010
Earlier work this paper cites.
Nearly unbiased variable selection under minimax concave penalty
Zhang, C.-H · 2010
Earlier work this paper cites.
Object detection with discriminatively trained part-based models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., and Ramanan, D · 2010
Earlier work this paper cites.
Self-paced learning for latent variable models
Kumar, M. P., Packer, B., and Koller, D · 2010
Earlier work this paper cites.
Nearly unbiased variable selection under minimax concave penalty
Zhang, C.-H · 2010
Earlier work this paper cites.
Robust statistics
Huber, P. J · 2011
Earlier work this paper cites.
How do humans teach: On curriculum learning and teaching dimension
Khan, F., Mutlu, B., and Zhu, X · 2011
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Imagenet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E · 2012
Earlier work this paper cites.
Self-paced learning for long-term tracking
Supancic, J. S. and Ramanan, D · 2013
Earlier work this paper cites.
A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems
Gong, P., Zhang, C., Lu, Z., Huang, J. Z., and Ye, J · 2013
Earlier work this paper cites.

Similar

Stochastic majorization-minimization algorithms for large-scale optimization
Mairal, J · 2013
Cited alongside, same era.
Self-paced learning with diversity
Jiang, L., Meng, D., Yu, S.-I., Lan, Z., Shan, S., and Hauptmann, A · 2014
Cited alongside, same era.
Training deep neural networks on noisy labels with bootstrapping
Original
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A · 2014
Cited alongside, same era.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Cited alongside, same era.
Training convolutional networks with noisy labels
Original
Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., and Fergus, R · 2014
Cited alongside, same era.
Self-paced learning with diversity
Jiang, L., Meng, D., Yu, S.-I., Lan, Z., Shan, S., and Hauptmann, A · 2014
Cited alongside, same era.
Training deep neural networks on noisy labels with bootstrapping
Original
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., and Rabinovich, A · 2014
Cited alongside, same era.
Dropout: a simple way to prevent neural networks from overfitting
Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R · 2014
Cited alongside, same era.
Webly supervised learning of convolutional networks
Chen, X. and Gupta, A · 2015
Cited alongside, same era.
Self-paced curriculum learning
Jiang, L., Meng, D., Zhao, Q., Shan, S., and Hauptmann, A. G · 2015
Cited alongside, same era.
What objective does self-paced learning indeed optimize?
Original
Meng, D., Zhao, Q., and Jiang, L · 2015
Cited alongside, same era.
Learning from corrupted binary labels via class-probability estimation
Menon, A., Van Rooyen, B., Ong, C. S., and Williamson, B · 2015
Cited alongside, same era.

Then

Rethinking the inception architecture for computer vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z · 2016
Later among the works it cites.
Wide residual networks
Zagoruyko, S. and Komodakis, N · 2016
Later among the works it cites.
A closer look at memorization in deep networks
Arpit, D., Jastrzkebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al · 2017
Closest in time.
Active bias: Training a more accurate neural network by emphasizing high variance samples
Chang, H.-S., Learned-Miller, E., and McCallum, A · 2017
Closest in time.
Avoiding your teacher’s mistakes: Training neural networks with controlled weak supervision
Original
Dehghani, M., Severyn, A., Rothe, S., and Kamps, J · 2017
Closest in time.
Self-paced learning: An implicit regularization perspective
Fan, Y., He, R., Liang, J., and Hu, B.-G · 2017
Closest in time.
Training deep neural-networks using a noise adaptation layer
Goldberger, J. and Ben-Reuven, E · 2017
Closest in time.
Automated curriculum learning for neural networks
Graves, A., Bellemare, M. G., Menick, J., Munos, R., and Kavukcuoglu, K · 2017
Closest in time.
Cleannet: Transfer learning for scalable image classifier training with label noise
Original
Lee, K.-H., He, X., Zhang, L., and Yang, L · 2017
Closest in time.
Exploring generalization in deep learning
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N · 2017
Closest in time.
Inception-v4, inception-resnet and the impact of residual connections on learning
Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A · 2017
Closest in time.
Toward robustness against label noise in training deep discriminative neural networks
Vahdat, A · 2017
Closest in time.
Learning from noisy large-scale datasets with minimal supervision
Veit, A., Alldrin, N., Chechik, G., Krasin, I., Gupta, A., and Belongie, S · 2017
Closest in time.
Robust probabilistic modeling with bayesian data reweighting
Wang, Y., Kucukelbir, A., and Blei, D. M · 2017
Closest in time.
A closer look at memorization in deep networks
Arpit, D., Jastrzkebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al · 2017
Closest in time.
Active bias: Training a more accurate neural network by emphasizing high variance samples
Chang, H.-S., Learned-Miller, E., and McCallum, A · 2017
Closest in time.
Training deep neural-networks using a noise adaptation layer
Goldberger, J. and Ben-Reuven, E · 2017
Closest in time.
Focal loss for dense object detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P · 2017
Closest in time.
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O · 2017
Closest in time.
Gradient descent with random initialization: Fast global convergence for nonconvex phase retrieval
Original
Chen, Y., Chi, Y., Fan, J., and Ma, C · 2018
Closest in time.
Fidelity-weighted learning
Dehghani, M., Mehrjou, A., Gouws, S., Kamps, J., and Schölkopf, B · 2018
Closest in time.
Learning to teach
Fan, Y., Tian, F., Qin, T., Li, X.-Y., and Liu, T.-Y · 2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…