Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

Understand

We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations.

• Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0\% sink rate consistently.
• The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps.
• Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions.