Energy Transformer

Understand

Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory.

• Attention is the power-house driving modern deep learning successes, but it lacks clear theoretical foundations.
• Energy-based models allow a principled approach to discriminative and generative tasks, but the design of the energy functional is not straightforward.
• At the same time, Dense Associative Memory models or Modern Hopfield Networks have a well-established theoretical foundation, and allow an intuitive design of the energy function.