Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences

Understand

Parallelizing Gated Recurrent Unit (GRU) networks is a challenging task, as the training procedure of GRU is inherently sequential.

• Prior efforts to parallelize GRU have largely focused on conventional parallelization strategies such as data-parallel and model-parallel training algorithms.
• However, when the given sequences are very long, existing approaches are still inevitably performance limited in terms of training time.
• In this paper, we present a novel parallel training scheme (called parallel-in-time) for GRU based on a multigrid reduction in time (MGRIT) solver.