Built on

Ernoult, M., Grollier, J., Querlioz, D., Bengio, Y. and Scellier, B., 2019. Updates of equilibrium prop match gradients of backprop through time in an RNN with static input. arXiv preprint arXiv:1905.13633 [v1]
Original
1905
Earlier work this paper cites.
Iuzzolino, M., Singer, Y. and Mozer, M.C., 2019. Convolutional bipartite attractor networks. arXiv preprint arXiv:1906.03504
Original
1906
Earlier work this paper cites.
Bai, S., Kolter, J.Z. and Koltun, V., 2019. Deep equilibrium models. arXiv preprint arXiv:1909.01377
Original
1909
Earlier work this paper cites.
Bartunov, S., Rae, J.W., Osindero, S. and Lillicrap, T.P., 2019. Meta-learning deep energy-based memory models. arXiv preprint arXiv:1910.02720
Original
1910
Earlier work this paper cites.
Ramon y Cajal, S., 1911. Histologie du système nerveux de l’homme et des vertébrés. Maloine, Paris, 2, pp.153-173
1911
Earlier work this paper cites.
Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), pp.2554-2558
1982
Earlier work this paper cites.
Hopfield, J.J., 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10), pp.3088-3092
1984
Earlier work this paper cites.
Fukushima, K., 1984. A hierarchical neural network model for associative memory. Biological cybernetics, 50(2), pp.105-113
1984
Earlier work this paper cites.
Pineda, F.J., 1987. Generalization of back-propagation to recurrent neural networks. Physical review letters, 59(19), p.2229
1987
Earlier work this paper cites.
Almeida, L.B., 1990. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In Artificial neural networks: concept learning (pp. 102-111)
1990
Earlier work this paper cites.
Felleman, D.J. and Van Essen, D.C., 1991. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1), pp.1-47
1991
Earlier work this paper cites.
Xie, X. and Seung, H.S., 2003. Equivalence of backpropagation and contrastive Hebbian learning in a layered network. Neural computation, 15(2), pp.441-454
2003
Earlier work this paper cites.

Similar

Hinton, G.E., Osindero, S. and Teh, Y.W., 2006. A fast learning algorithm for deep belief nets. Neural computation, 18(7), pp.1527-1554
2006
Cited alongside, same era.
Hinton, G.E. and Salakhutdinov, R.R., 2006. Reducing the dimensionality of data with neural networks. science, 313(5786), pp.504-507
2006
Cited alongside, same era.
Widrich, M., Schäfl, B., Ramsauer, H., Pavlović, M., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, G.K., Greiff, V., Hochreiter, S. and Klambauer, G., 2020. Modern Hopfield networks and attention for immune repertoire classification. arXiv preprint arXiv:2007.13505
Original
2007
Cited alongside, same era.
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G.K., Greiff, V. and Kreil, D., 2020. Hopfield Networks is All You Need. arXiv preprint arXiv:2008.02217
Original
2008
Cited alongside, same era.

Then

Chen, R.T., Rubanova, Y., Bettencourt, J. and Duvenaud, D., 2018. Neural ordinary differential equations. arXiv preprint arXiv:1806.07366
Original
2018
Later among the works it cites.
Barra, A., Beccaria, M. and Fachechi, A., 2018. A new mechanical approach to handle generalized Hopfield neural networks. Neural Networks, 106, pp.205-222
2018
Later among the works it cites.
Krotov, D. and Hopfield, J., 2018. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12), pp.3151-3167
2018
Later among the works it cites.
Cembrowski, M.S. and Spruston, N., 2019. Heterogeneity within classical cell types is the rule: lessons from hippocampal pyramidal neurons. Nature Reviews Neuroscience, 20(4), pp.193-204
2019
Later among the works it cites.
Agliari, E. and De Marzo, G., 2020. Tolerance versus synaptic noise in dense associative memories. The European Physical Journal Plus, 135(11), pp.1-22

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…

2021

Hierarchical Associative Memory

Krotov, Dmitry

Understand

Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory.

They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons.
At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications.
This paper tackles this gap and describes a fully recurrent model of associative memory with an arbitrary large number of layers, some of which can be locally connected (convolutional), and a corresponding energy function that decreases on the dynamical trajectory of the neurons' activations.

Built on

Ernoult, M., Grollier, J., Querlioz, D., Bengio, Y. and Scellier, B., 2019. Updates of equilibrium prop match gradients of backprop through time in an RNN with static input. arXiv preprint arXiv:1905.13633 [v1]
Original
1905
Earlier work this paper cites.
Iuzzolino, M., Singer, Y. and Mozer, M.C., 2019. Convolutional bipartite attractor networks. arXiv preprint arXiv:1906.03504
Original
1906
Earlier work this paper cites.
Bai, S., Kolter, J.Z. and Koltun, V., 2019. Deep equilibrium models. arXiv preprint arXiv:1909.01377
Original
1909
Earlier work this paper cites.
Bartunov, S., Rae, J.W., Osindero, S. and Lillicrap, T.P., 2019. Meta-learning deep energy-based memory models. arXiv preprint arXiv:1910.02720
Original
1910
Earlier work this paper cites.
Ramon y Cajal, S., 1911. Histologie du système nerveux de l’homme et des vertébrés. Maloine, Paris, 2, pp.153-173
1911
Earlier work this paper cites.
Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), pp.2554-2558
1982
Earlier work this paper cites.
Hopfield, J.J., 1984. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10), pp.3088-3092
1984
Earlier work this paper cites.
Fukushima, K., 1984. A hierarchical neural network model for associative memory. Biological cybernetics, 50(2), pp.105-113
1984
Earlier work this paper cites.
Pineda, F.J., 1987. Generalization of back-propagation to recurrent neural networks. Physical review letters, 59(19), p.2229
1987
Earlier work this paper cites.
Almeida, L.B., 1990. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. In Artificial neural networks: concept learning (pp. 102-111)
1990
Earlier work this paper cites.
Felleman, D.J. and Van Essen, D.C., 1991. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1), pp.1-47
1991
Earlier work this paper cites.
Xie, X. and Seung, H.S., 2003. Equivalence of backpropagation and contrastive Hebbian learning in a layered network. Neural computation, 15(2), pp.441-454
2003
Earlier work this paper cites.

Similar

Hinton, G.E., Osindero, S. and Teh, Y.W., 2006. A fast learning algorithm for deep belief nets. Neural computation, 18(7), pp.1527-1554
2006
Cited alongside, same era.
Hinton, G.E. and Salakhutdinov, R.R., 2006. Reducing the dimensionality of data with neural networks. science, 313(5786), pp.504-507
2006
Cited alongside, same era.
Widrich, M., Schäfl, B., Ramsauer, H., Pavlović, M., Gruber, L., Holzleitner, M., Brandstetter, J., Sandve, G.K., Greiff, V., Hochreiter, S. and Klambauer, G., 2020. Modern Hopfield networks and attention for immune repertoire classification. arXiv preprint arXiv:2007.13505
Original
2007
Cited alongside, same era.
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G.K., Greiff, V. and Kreil, D., 2020. Hopfield Networks is All You Need. arXiv preprint arXiv:2008.02217
Original
2008
Cited alongside, same era.

Then

Chen, R.T., Rubanova, Y., Bettencourt, J. and Duvenaud, D., 2018. Neural ordinary differential equations. arXiv preprint arXiv:1806.07366
Original
2018
Later among the works it cites.
Barra, A., Beccaria, M. and Fachechi, A., 2018. A new mechanical approach to handle generalized Hopfield neural networks. Neural Networks, 106, pp.205-222
2018
Later among the works it cites.
Krotov, D. and Hopfield, J., 2018. Dense associative memory is robust to adversarial inputs. Neural computation, 30(12), pp.3151-3167
2018
Later among the works it cites.
Cembrowski, M.S. and Spruston, N., 2019. Heterogeneity within classical cell types is the rule: lessons from hippocampal pyramidal neurons. Nature Reviews Neuroscience, 20(4), pp.193-204
2019
Later among the works it cites.
Agliari, E. and De Marzo, G., 2020. Tolerance versus synaptic noise in dense associative memories. The European Physical Journal Plus, 135(11), pp.1-22

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Krotov, D. and Hopfield, J., 2020. Large Associative Memory Problem in Neurobiology and Machine Learning. arXiv preprint arXiv:2008.06996

Original

2008

Cited alongside, same era.

Tang, F. and Kopp, M., 2021. A remark on a paper of Krotov and Hopfield [arXiv: 2008.06996]. arXiv preprint arXiv:2105.15034

Original

2008

Cited alongside, same era.

Salakhutdinov, R. and Hinton, G., 2009, April. Deep boltzmann machines. In Artificial intelligence and statistics (pp. 448-455). PMLR

2009

Cited alongside, same era.

Krotov, D. and Hopfield, J.J., 2016. Dense associative memory for pattern recognition. In Advances in neural information processing systems (pp. 1172-1180), arXiv:1606.01164

Original

2016

Cited alongside, same era.

Dumoulin, V. and Visin, F., 2016. A guide to convolution arithmetic for deep learning. arXiv preprint arXiv:1603.07285

Original

2016

Cited alongside, same era.

Demircigil, M., Heusel, J., Löwe, M., Upgang, S. and Vermet, F., 2017. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2), pp.288-299

2017

Cited alongside, same era.

Scellier, B. and Bengio, Y., 2017. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11, p.24

2017

Cited alongside, same era.

2020

Later among the works it cites.

Agliari, E., Alemanno, F., Barra, A. and Fachechi, A., 2020. Generalized Guerra’s interpolation schemes for dense associative neural networks. Neural Networks, 128, pp.254-267

2020

Later among the works it cites.

Marullo, C. and Agliari, E., 2021. Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks. Entropy, 23(1), p.34

2021

Closest in time.

Kirchberger, L., Mukherjee, S., Schnabel, U.H., van Beest, E.H., Barsegyan, A., Levelt, C.N., Heimel, J.A., Lorteije, J.A., van der Togt, C., Self, M.W. and Roelfsema, P.R., 2021. The essential role of recurrent processing for figure-ground perception in mice. Science Advances, 7(27), p.eabe1833

2021

Closest in time.

Nguyen, N. and Chang, J.M., 2021. COVID-19 Pneumonia Severity Prediction using Hybrid Convolution-Attention Neural Architectures. arXiv preprint arXiv:2107.02672

Original

2021

Closest in time.

Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Keysers, D., Uszkoreit, J., Lucic, M. and Dosovitskiy, A., 2021. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2105.01601

Original

2021

Closest in time.

Melas-Kyriazi, L., 2021. Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet. arXiv preprint arXiv:2105.02723

Original

2021

Closest in time.