Statistical Characteristics of Deep Representations: An Empirical Investigation

Understand

In this study, the effects of eight representation regularization methods are investigated, including two newly developed rank regularizers (RR).

• The investigation shows that the statistical characteristics of representations such as correlation, sparsity, and rank can be manipulated as intended, during training.
• Furthermore, it is possible to improve the baseline performance simply by trying all the representation regularizers and fine-tuning the strength of their effects.
• In contrast to performance improvement, no consistent relationship between performance and statistical characteristics was observable.