NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration

Understand

We present NetReduce, a novel RDMA-compatible in-network reduction architecture to accelerate distributed DNN training.

• Compared to existing designs, NetReduce maintains a reliable connection between end-hosts in the Ethernet and does not terminate the connection in the network.
• The advantage of doing so is that we can fully reuse the designs of congestion control and reliability in RoCE.
• In the meanwhile, we do not need to implement a high-cost network protocol processing stack in the switch, as IB does.