Understand
We introduce the "inverse square root linear unit" (ISRLU) to speed up learning in deep neural networks.
- ISRLU has better performance than ELU but has many of the same benefits.
- ISRLU and ELU have similar curves and characteristics.
- Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noise-robust deactivation state, lessening the over fitting risk.
Built on
Arithmetic Complexity of Computations , volume 33 of CBMS-NSF Regional Conference Series in Applied Mathematics
Shmuel Winograd · 1980
Earlier work this paper cites.
Programming the FPS T Series
John L. Gustafson · 1986
Earlier work this paper cites.
Algorithms and design: the CRAY APP shared-memory system
Brad Carlile · 1993
Earlier work this paper cites.
Long short-term memory
Sepp Hochreiter and Jürgen Schmidhuber · 1997
Earlier work this paper cites.
Natural gradient works efficiently in learning
Shun-Ichi Amari · 1998
Earlier work this paper cites.
Fast inverse square root
Chris Lomont · 2003
Earlier work this paper cites.
Similar
Deep sparse rectifier neural networks
Xavier Glorot, Antoine Bordes, and Yoshua Bengio · 2011
Cited alongside, same era.
Empirical evaluation of gated recurrent neural networks on sequence modeling
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio · 2014
Cited alongside, same era.
Fast and accurate deep network learning by exponential linear units (ELUs)
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter · 2015
Cited alongside, same era.
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun · 2015
Cited alongside, same era.
Tensorflow: A system for large-scale machine learning
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng · 2016
Cited alongside, same era.
Parallelism, compute intensity, and data vectorization
Brad Carlile
Cited in the paper.
Then
Fast algorithms for convolutional neural networks
Andrew Lavin and Scott Gray · 2016
Later among the works it cites.
Deep residual networks with exponential linear unit
Anish Shah, Eashan Kadam, Hena Shah, Sameer Shinde, and Sandip Shingade · 2016
Later among the works it cites.
Rethinking the inception architecture for computer vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna · 2016
Later among the works it cites.
Intel, Sep 2017
Vector Mathematics (VM) Performance and Accuracy Data · 2017
Closest in time.
Self-normalizing neural networks
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter · 2017
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…