Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results

Understand

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks.

• It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target.
• However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets.
• To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions.