Towards Memory-Efficient Training for Extremely Large Output Spaces -- Learning with 500k Labels on a Single Commodity GPU

Understand

In classification problems with large output spaces (up to millions of labels), the last layer can require an enormous amount of memory.

• Using sparse connectivity would drastically reduce the memory requirements, but as we show below, it can result in much diminished predictive performance of the model.
• Fortunately, we found that this can be mitigated by introducing a penultimate layer of intermediate size.
• We further demonstrate that one can constrain the connectivity of the sparse layer to be uniform, in the sense that each output neuron will have the exact same number of incoming connections.