ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients

Understand

Neural Architecture Search (NAS) is widely used to automatically obtain the neural network with the best performance among a large number of candidate architectures.

• To reduce the search time, zero-shot NAS aims at designing training-free proxies that can predict the test performance of a given architecture.
• However, as shown recently, none of the zero-shot proxies proposed to date can actually work consistently better than a naive proxy, namely, the number of network parameters (#Params).
• To improve this state of affairs, as the main theoretical contribution, we first reveal how some specific gradient properties across different samples impact the convergence rate and generalization capacity of neural networks.