RSO: A Gradient Free Sampling Based Approach For Training Deep Neural Networks

Understand

We propose RSO (random search optimization), a gradient free Markov Chain Monte Carlo search based approach for training deep neural networks.

• To this end, RSO adds a perturbation to a weight in a deep neural network and tests if it reduces the loss on a mini-batch.
• If this reduces the loss, the weight is updated, otherwise the existing weight is retained.
• Surprisingly, we find that repeating this process a few times for each weight is sufficient to train a deep neural network.