NAS evaluation is frustratingly hard

Understand

Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012.

• Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue.
• While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all.
• As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others.