Bypass Exponential Time Preprocessing: Fast Neural Network Training via Weight-Data Correlation Preprocessing

Understand

Over the last decade, deep neural networks have transformed our society, and they are already widely applied in various machine learning applications.

• State-of-art deep neural networks are becoming larger in size every year to deliver increasing model accuracy, and as a result, model training consumes substantial computing resources and will only consume more in the future.
• Using current training methods, in each iteration, to process a data point $x \in \mathbb{R}^d$ in a layer, we need to spend $\Theta(md)$ time to evaluate all the $m$ neurons in the layer.
• This means processing the entire layer takes $\Theta(nmd)$ time for $n$ data points.