Self-attention as an attractor network: transient memories without backpropagation

Deep Equilibrium Models, October 2019

Original

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun · 1909

Earlier work this paper cites.

Statistical Analysis of Non-Lattice Data

Julian Besag · 1975

Earlier work this paper cites.

Hopfield Networks is All You Need, April 2021

Original

Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter · 2008

Earlier work this paper cites.

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021

Original

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby · 2010

Earlier work this paper cites.

Layer normalization, 2016

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton · 2016

Earlier work this paper cites.

Attention is All you Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin · 2017

Earlier work this paper cites.

On a Model of Associative Memory with Huge Storage Capacity

Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet · 2017

Earlier work this paper cites.

Inverse statistical problems: from the inverse Ising problem to data science

H. Chau Nguyen, Riccardo Zecchina, and Johannes Berg · 2017

Cited alongside, same era.

Overparameterized neural networks implement associative memory

Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler · 2020

Cited alongside, same era.

Highly accurate protein structure prediction with AlphaFold

John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis · 2021

Cited alongside, same era.

Deep implicit attention: A mean-field theory perspective on attention mechanisms, May 2021

Matthias Bal · 2021

Cited alongside, same era.

Transformers are secretly collectives of spin systems, November 2021

Matthias Bal · 2021

Transformers from an Optimization Perspective, February 2023

Original

Yongyi Yang, Zengfeng Huang, and David Wipf · 2023

Later among the works it cites.

Spin-model transformers, December 2023

Matthias Bal · 2023

Later among the works it cites.

Mapping of attention mechanisms to a generalized Potts model

Riccardo Rende, Federica Gerace, Alessandro Laio, and Sebastian Goldt · 2024

Closest in time.

Exponential Capacity of Dense Associative Memories

Carlo Lucibello and Marc Mézard · 2024

Closest in time.

Random Features Hopfield Networks generalize retrieval to previously unseen examples, July 2024

Original

Silvio Kalaj, Clarissa Lauditi, Gabriele Perugini, Carlo Lucibello, Enrico M. Malatesta, and Matteo Negri · 2024

Closest in time.