Deeper Insights into Weight Sharing in Neural Architecture Search

Understand

With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention.

• As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure.
• These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model.
• However, weight-sharing has no theoretical guarantee and its impact has not been well studied before.