Fetching the paper…
Reading the bibliography…
The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks.
Householder, Alston S. “A Theory of Steady-State Activity in Nerve-Net.” The Bulletin of Mathematical Biophysics, vol. 3, no. 2, 1941, pp. 63-69
1941
Earlier work this paper cites.
Fukushima, Kunihiko. “Visual Feature Extraction by a Multilayered Network of Analog Threshold Elements.” IEEE Transactions on Systems Science and Cybernetics, vol. 5, no. 4, 1969, pp. 322-333
1969
Earlier work this paper cites.
Rumelhart, David E., Geoffrey E. Hinton, and Ronald J. Williams. Learning internal representations by error propagation. No. ICS-8506. California Univ San Diego La Jolla Inst for Cognitive Science, 1985
1985
Earlier work this paper cites.
LeCun, Yann, et al. “Gradient-based learning applied to document recognition.” Proceedings of the IEEE 86.11 (1998): 2278-2324
1998
Earlier work this paper cites.
Hahnloser, Richard HR, et al. “Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.” Nature 405.6789 (2000): 947
2000
Earlier work this paper cites.
Hochreiter, Sepp, et al. “Gradient flow in recurrent nets: the difficulty of learning long-term dependencies.” (2001)
2001
Earlier work this paper cites.
Deng, Jia, et al. “Imagenet: A large-scale hierarchical image database.” 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009
2009
Earlier work this paper cites.
Krizhevsky, Alex, and Geoffrey Hinton. “Learning multiple layers of features from tiny images.” (2009): 7
2009
Earlier work this paper cites.
Glorot, Xavier, and Yoshua Bengio. “Understanding the difficulty of training deep feedforward neural networks.” Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010
2010
Earlier work this paper cites.
Nair, Vinod, and Geoffrey E. Hinton. “Rectified linear units improve restricted boltzmann machines.” Proceedings of the 27th international conference on machine learning (ICML-10). 2010
2010
Cited alongside, same era.
Glorot, Xavier, Antoine Bordes, and Yoshua Bengio. “Deep Sparse Rectifier Neural Networks.” Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011, pp. 315-323
2011
Cited alongside, same era.
Maas, Andrew L., et al. “Learning word vectors for sentiment analysis.” Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1. Association for Computational Linguistics, 2011
2011
Cited alongside, same era.
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “Imagenet classification with deep convolutional neural networks.” Advances in neural information processing systems. 2012
2012
Cited alongside, same era.
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016
2016
Later among the works it cites.
He, Kaiming, et al. “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2016
2016
Later among the works it cites.
2016
Later among the works it cites.
2017
Later among the works it cites.
Zhu, Jun-Yan, et al. “Unpaired image-to-image translation using cycle-consistent adversarial networks.” Proceedings of the IEEE international conference on computer vision. 2017
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…
Maas, Andrew L., Awni Y. Hannun, and Andrew Y. Ng. “Rectifier nonlinearities improve neural network acoustic models.” Proc. icml. Vol. 30. No. 1. 2013
2013
Cited alongside, same era.
Goodfellow, Ian, et al. “Generative adversarial nets.” Advances in neural information processing systems. 2014
2014
Cited alongside, same era.
2015
Cited alongside, same era.
Vinyals, Oriol, et al. “Show and tell: A neural image caption generator.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2015
2015
Cited alongside, same era.
Xu, Kelvin, et al. “Show, attend and tell: Neural image caption generation with visual attention.” International conference on machine learning. 2015
2015
Cited alongside, same era.
2017
Later among the works it cites.
Devlin, Jacob, et al. “Bert: Pre-training of deep bidirectional transformers for language understanding.” Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019
2019
Closest in time.
Paszke, Adam, et al. “Pytorch: An imperative style, high-performance deep learning library.” Advances in neural information processing systems 32 (2019)
2019
Closest in time.
Smith, Leslie N., and Nicholay Topin. “Super-convergence: Very fast training of neural networks using large learning rates.” Artificial intelligence and machine learning for multi-domain operations applications. Vol. 11006. SPIE, 2019
2019
Closest in time.