SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks

Understand

Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models.

• There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g.
• depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models.
• To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs.