2017

Be Careful What You Backpropagate: A Case For Linear Output Activations & Gradient Boosting

Oland, Anders, Bansal, Aayush, Dannenberg, Roger B. et al.

Understand

In this work, we show that saturating output activation functions, such as the softmax, impede learning on a number of standard classification tasks.

Moreover, we present results showing that the utility of softmax does not stem from the normalization, as some have speculated.
In fact, the normalization makes things worse.
Rather, the advantage is in the exponentiation of error gradients.

Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986) · 1986
Earlier work this paper cites.
Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition
Bridle, J. S. (1990) · 1990
Earlier work this paper cites.
Untersuchungen zu dynamischen neuronalen Netzen
Hochreiter, S. (1991) · 1991
Earlier work this paper cites.
Neural Networks for Pattern Recognition
Bishop, C. M. (1995) · 1995
Earlier work this paper cites.
Gradient Based Learning Applied to Document Recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998) · 1998
Earlier work this paper cites.
Pattern Recognition and Machine Learning (Information Science and Statistics), 1st edn. 2006. corr. 2nd printing edn
Bishop, C. (2007) · 2006
Earlier work this paper cites.
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., and LeCun, Y. (2009) · 2009
Earlier work this paper cites.
Learning multiple layers of features from tiny images
Krizhevsky, A. (2009) · 2009
Earlier work this paper cites.
The PASCAL Visual Object Classes (VOC) Challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. (2010) · 2010
Earlier work this paper cites.

Understanding the difficulty of training deep feedforward neural networks
Glorot, X. and Bengio, Y. (2010) · 2010
Cited alongside, same era.
Rectified linear units improve restricted boltzmann machines
Nair, V. and Hinton, G. E. (2010) · 2010
Cited alongside, same era.
Semantic contours from inverse detectors
Hariharan, B., Arbel�ez, P., Bourdev, L., Maji, S., and Malik, J. (2011) · 2011
Cited alongside, same era.
Improving neural networks by preventing co-adaptation of feature detectors
Original
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. (2012) · 2012
Cited alongside, same era.
Feedforward Inhibition and Synaptic Scaling – Two Sides of the Same Coin?
Keck, C., Savin, C., Lücke, J., King, A., and Ros, H. (2012) · 2012
Cited alongside, same era.
ImageNet Classification with Deep Convolutional Neural Networks
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012) · 2012
Cited alongside, same era.
Efficient BackProp
LeCun, Y. A., Bottou, L., Orr, G. B., and Müller, K. R. (2012) · 2012
Cited alongside, same era.
Very deep convolutional networks for large-scale image recognition
Original
Simonyan, K. and Zisserman, A. (2014) · 2014
Cited alongside, same era.
Striving for simplicity: The all convolutional net
Original
Springenberg, J. T., Dosovitskiy, A., Brox, T., and Riedmiller, M. A. (2014) · 2014
Cited alongside, same era.

Hypercolumns for object segmentation and fine-grained localization
Hariharan, B., Arbeláez, P., Girshick, R., and Malik, J. (2015) · 2015
Later among the works it cites.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Original
Ioffe, S. and Szegedy, C. (2015) · 2015
Later among the works it cites.
Fully convolutional models for semantic segmentation
Long, J., Shelhamer, E., and Darrell, T. (2015) · 2015
Later among the works it cites.
ImageNet large scale visual recognition challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015) · 2015
Later among the works it cites.
Designing deep networks for surface normal estimation
Wang, X., Fouhey, D., and Gupta, A. (2015) · 2015
Later among the works it cites.
Deep learning
Goodfellow, I., Bengio, Y., and Courville, A. (2016) · 2016
Later among the works it cites.
Pixelnet: Representation of the pixels, by the pixels, and for the pixels
Original
Bansal, A., Chen, X., Russell, B., Gupta, A., and Ramanan, D. (2017) · 2017
Closest in time.
Learning from Synthetic Humans
Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M. J., Laptev, I., and Schmid, C. (2017) · 2017
Closest in time.

alphaXiv searches the wider corpus for related work and actual follow-ups.

alphaXiv is searching for related work…

Reading the bibliography…