Nondeterminism and Instability in Neural Network Optimization

Understand

Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability.

• While uncertainty can be reduced by training multiple model copies, doing so is time-consuming, costly, and harms reproducibility.
• In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, allowing us to isolate the effects of a variety of sources of nondeterminism.
• Surprisingly, we find that all sources of nondeterminism have similar effects on measures of model diversity.