Out-of-core Training for Extremely Large-Scale Neural Networks With Adaptive Window-Based Scheduling

Understand

While large neural networks demonstrate higher performance in various tasks, training large networks is difficult due to limitations on GPU memory size.

• We propose a novel out-of-core algorithm that enables faster training of extremely large-scale neural networks with sizes larger than allotted GPU memory.
• Under a given memory budget constraint, our scheduling algorithm locally adapts the timing of memory transfers according to memory usage of each function, which improves overlap between computation and memory transfers.
• Additionally, we apply virtual addressing technique, commonly performed in OS, to training of neural networks with out-of-core execution, which drastically reduces the amount of memory fragmentation caused by frequent memory transfers.