How far can we go without convolution: Improving fully-connected networks

Understand

We propose ways to improve the performance of fully connected networks.

• We found that two approaches in particular have a strong effect on performance: linear bottleneck layers and unsupervised pre-training using autoencoders without hidden unit biases.
• We show how both approaches can be related to improving gradient flow and reducing sparsity in the network.
• We show that a fully connected network can yield approximately 70% classification accuracy on the permutation-invariant CIFAR-10 task, which is much higher than the current state-of-the-art.