Optimal Gradient Checkpoint Search for Arbitrary Computation Graphs

Understand

Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases.

• Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance.
• Unlike solutions that require physically upgrade GPUs, the Gradient CheckPointing(GCP) training trades computation for more memory beyond existing GPU hardware.
• GCP only stores a subset of intermediate tensors, called Gradient Checkpoints (GCs), during forward.