IGLU: Efficient GCN Training via Lazy Updates

Understand

Training multi-layer Graph Convolution Networks (GCN) using standard SGD techniques scales poorly as each descent step ends up updating node embeddings for a large portion of the graph.

• Recent attempts to remedy this sub-sample the graph that reduces compute but introduce additional variance and may offer suboptimal performance.
• This paper develops the IGLU method that caches intermediate computations at various GCN layers thus enabling lazy updates that significantly reduce the compute cost of descent.
• IGLU introduces bounded bias into the gradients but nevertheless converges to a first-order saddle point under standard assumptions such as objective smoothness.