Avoiding Your Teacher's Mistakes: Training Neural Networks with Controlled Weak Supervision

Understand

Training deep neural networks requires massive amounts of training data, but for many tasks only limited labeled data is available.

• This makes weak supervision attractive, using weak or noisy signals like the output of heuristic methods or user click-through data for training.
• In a semi-supervised setting, we can use a large set of data with weak labels to pretrain a neural network and then fine-tune the parameters with a small amount of data with true labels.
• This feels intuitively sub-optimal as these two independent stages leave the model unaware about the varying label quality.