Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes

Understand

The training dynamics of linear networks are well studied in two distinct setups: the lazy regime and balanced/active regime, depending on the initialization and width of the network.

• We provide a surprisingly simple unifying formula for the evolution of the learned matrix that contains as special cases both lazy and balanced regimes but also a mixed regime in between the two.
• In the mixed regime, a part of the network is lazy while the other is balanced.
• More precisely the network is lazy along singular values that are below a certain threshold and balanced along those that are above the same threshold.