WLD-Reg: A Data-dependent Within-layer Diversity Regularizer

Understand

Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization, where the errors are back-propagated from the last layer back to the first one.

• At each optimization step, neurons at a given layer receive feedback from neurons belonging to higher layers of the hierarchy.
• In this paper, we propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer.
• To this end, we measure the pairwise similarity between the outputs of the neurons and use it to model the layer's overall diversity.