Efficient and Sparse Neural Networks by Pruning Weights in a Multiobjective Learning Approach

Understand

Overparameterization and overfitting are common concerns when designing and training deep neural networks, that are often counteracted by pruning and regularization strategies.

• However, these strategies remain secondary to most learning approaches and suffer from time and computational intensive procedures.
• We suggest a multiobjective perspective on the training of neural networks by treating its prediction accuracy and the network complexity as two individual objective functions in a biobjective optimization problem.
• As a showcase example, we use the cross entropy as a measure of the prediction accuracy while adopting an l1-penalty function to assess the total cost (or complexity) of the network parameters.