From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

Understand

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities.

• After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation.
• Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss.
• We reveal an unexpected connection between this new loss and the Huber classification loss.