signProx: One-Bit Proximal Algorithm for Nonconvex Stochastic Optimization

Understand

Stochastic gradient descent (SGD) is one of the most widely used optimization methods for parallel and distributed processing of large datasets.

• One of the key limitations of distributed SGD is the need to regularly communicate the gradients between different computation nodes.
• To reduce this communication bottleneck, recent work has considered a one-bit variant of SGD, where only the sign of each gradient element is used in optimization.
• In this paper, we extend this idea by proposing a stochastic variant of the proximal-gradient method that also uses one-bit per update element.