2018

Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise

Hendrycks, Dan, Mazeika, Mantas, Wilson, Duncan et al.

Understand

The growing importance of massive datasets used for deep learning makes robustness to label noise a critical property for classifiers to have.

Sources of label noise include automatic labeling, non-expert labeling, and label corruption by data poisoning adversaries.
Numerous previous works assume that no source of labels can be trusted.
We relax this assumption and assume that a small subset of the training data is trusted.

“Design of robust neural network classifiers”
J Larsen, L Nonboe, M Hintz-Madsen and L Hansen · 1998
Earlier work this paper cites.
“Class Noise vs. Attribute Noise: A Quantitative Study”
Xingquan Zhu and Xindong Wu · 2004
Earlier work this paper cites.
“Class Noise and Supervised Learning in Medical Domains: The Effect of Feature Extraction”
M Pechenizkiy, A Tsymbal, S Puuronen and O Pechenizkiy · 2006
Earlier work this paper cites.
“A study of the effect of different types of noise on the precision of supervised learning techniques”
David Nettleton, Albert Orriols-Puig and Albert Fornells · 2010
Earlier work this paper cites.
“Support Vector Machines Under Adversarial Label Noise”
B Biggio, B Nelson and P Laskov · 2011
Earlier work this paper cites.
“Part-of-speech Tagging for Twitter: Annotation, Features, and Experiments”
Kevin Gimpel et al · 2011
Earlier work this paper cites.
“Learning Word Vectors for Sentiment Analysis”
Andrew. Maas et al · 2011
Earlier work this paper cites.
“Learning to label aerial images from noisy data”
Volodymyr Mnih and Geoffrey Hinton · 2012
Earlier work this paper cites.
“Learning with Noisy Labels”
Nagarajan Natarajan, Inderjit Dhillon, Pradeep Ravikumar and Ambuj Tewari · 2013
Earlier work this paper cites.
“Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank”
Richard Socher et al · 2013
Earlier work this paper cites.

“Classification in the presence of label noise: a survey”
Beno\ˆt Fr\’enay and Michel Verleysen · 2014
Cited alongside, same era.
“Adam: A Method for Stochastic Optimization”
Diederik. Kingma and Jimmy Ba · 2014
Cited alongside, same era.
“Training Deep Neural Networks on Noisy Labels with Bootstrapping”
Scott Reed et al · 2014
Cited alongside, same era.
“Dropout: A simple way to prevent neural networks from overfitting”
Nitish Srivastava et al · 2014
Cited alongside, same era.
“Training Convolutional Networks with Noisy Labels”
Sainbayar Sukhbaatar et al · 2014
Cited alongside, same era.
“Learning from massive noisy labeled data for image classification”
Tong Xiao et al · 2015
Cited alongside, same era.
“Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units”
Dan Hendrycks and Kevin Gimpel · 2016
Cited alongside, same era.
“Data Poisoning Attacks on Factorization-Based Collaborative Filtering”
Bo Li, Yining Wang, Aarti Singh and Yevgeniy Vorobeychik · 2016
Cited alongside, same era.
“SGDR: Stochastic Gradient Descent with Warm Restarts”
Ilya Loshchilov and Frank Hutter · 2016
Cited alongside, same era.
“Learning from Binary Labels with Instance-Dependent Corruption”
Aditya Menon, Brendan van Rooyen and Nagarajan Natarajan · 2016
Cited alongside, same era.

“Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach”
Giorgio Patrini et al · 2016
Later among the works it cites.
“Aggregated Residual Transformations for Deep Neural Networks”
Saining Xie et al · 2016
Later among the works it cites.
“Wide Residual Networks”
Sergey Zagoruyko and Nikos Komodakis · 2016
Later among the works it cites.
“Learning from Untrusted Data”
Moses Charikar, Jacob Steinhardt and Gregory Valiant · 2017
Later among the works it cites.
“On Calibration of Modern Neural Networks”
Chuan Guo, Geoff Pleiss, Yu Sun and Kilian. Weinberger · 2017
Later among the works it cites.
“Learning from Noisy Labels with Distillation”
Yuncheng Li et al · 2017
Later among the works it cites.
“Certified Defenses for Data Poisoning Attacks”
Jacob Steinhardt, Pang Koh and Percy Liang · 2017
Later among the works it cites.
“Learning From Noisy Large-Scale Datasets With Minimal Supervision”
Andreas Veit et al · 2017
Later among the works it cites.
“Learning to Reweight Examples for Robust Deep Learning”
Thu Ren, Wenyuan Zeng, Bin Yang and Raquel Urtasun · 2018
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…