GADGET: Online Resource Optimization for Scheduling Ring-All-Reduce Learning Jobs

Understand

Fueled by advances in distributed deep learning (DDL), recent years have witnessed a rapidly growing demand for resource-intensive distributed/parallel computing to process DDL computing jobs.

• To resolve network communication bottleneck and load balancing issues in distributed computing, the so-called ``ring-all-reduce'' decentralized architecture has been increasingly adopted to remove the need for dedicated parameter servers.
• To date, however, there remains a lack of theoretical understanding on how to design resource optimization algorithms for efficiently scheduling ring-all-reduce DDL jobs in computing clusters.
• This motivates us to fill this gap by proposing a series of new resource scheduling designs for ring-all-reduce DDL jobs.