Understand
Softmax function is widely used in artificial neural networks for multiclass classification, multilabel classification, attention mechanisms, etc.
- However, its efficacy is often questioned in literature.
- The log-softmax loss has been shown to belong to a more generic class of loss functions, called spherical family, and its member log-Taylor softmax loss is arguably the best alternative in this class.
- In another approach which tries to enhance the discriminative nature of the softmax function, soft-margin softmax (SM-softmax) has been proposed to be the most suitable alternative.
Built on
“ImageNet Large Scale Visual Recognition Challenge”
Olga Russakovsky et al · 2015
Earlier work this paper cites.
“Efficient Exact Gradient Update for training Deep Networks with Very Large Sparse Targets”
Pascal Vincent, Alexandre de Brébisson and Xavier Bouthillier · 2015
Earlier work this paper cites.
“An Exploration of Softmax Alternatives Belonging to the Spherical Loss Family”
Alexandre de Brébisson and Pascal Vincent · 2016
Earlier work this paper cites.
“Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)”
Djork-Arné Clevert, Thomas Unterthiner and Sepp Hochreiter · 2016
Earlier work this paper cites.
Similar
“Large-Margin Softmax Loss for Convolutional Neural Networks”
Weiyang Liu, Yandong Wen, Zhiding Yu and Meng Yang · 2016
Cited alongside, same era.
“Soft-Margin Softmax for Deep Classification”
Xuezhi Liang et al · 2017
Cited alongside, same era.
“Attention is All you Need”
Ashish Vaswani et al · 2017
Cited alongside, same era.
Then
“DropMax: Adaptive Variationial Softmax”
Hae Lee et al · 2018
Later among the works it cites.
“How to Develop a CNN From Scratch for CIFAR-10 Photo Classification” Accessed: 2020-06-21, https://machinelearningmastery.com/how-to-develop-a-cnn-from-scratch-for-cifar-10-photo-classification/
Jason Brownlee · 2020
Closest in time.
“Show, Attend and Tell: Neural Image Caption Generation with Visual Attention”
Kelvin Xu et al · 2057
Closest in time.
Beyond the bibliography
alphaXiv searches the wider corpus for related work and actual follow-ups.
alphaXiv is searching for related work…