2016

Dense Associative Memory for Pattern Recognition

Krotov, Dmitry, Hopfield, John J

Understand

A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network.

We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning.
On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed.
One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime.

Built on

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), pp.1929-1958
1958
Earlier work this paper cites.
Minsky, M. and Papert, S., 1969. Perceptron: an introduction to computational geometry. The MIT Press, Cambridge, expanded edition, 19(88), p.2
1969
Earlier work this paper cites.
Hopfield, J.J., 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8), pp.2554-2558
1982
Earlier work this paper cites.
Amit, D.J., Gutfreund, H. and Sompolinsky, H., 1985. Storing infinite numbers of patterns in a spin-glass model of neural networks. Physical Review Letters, 55(14), p.1530
1985
Earlier work this paper cites.
Chen, H.H., Lee, Y.C., Sun, G.Z., Lee, H.Y., Maxwell, T. and Giles, C.L., 1986. High order correlation model for associative memory. In Neural Networks for Computing (Vol. 151, No. 1, pp. 86-99). AIP Publishing
1986
Earlier work this paper cites.
Psaltis, D. and Park, C.H., 1986. Nonlinear discriminant functions and associative memories. In Neural networks for computing (Vol. 151, No. 1, pp. 370-375). AIP Publishing
1986
Earlier work this paper cites.
McEliece, R.J., Posner, E.C., Rodemich, E.R. and Venkatesh, S.S., 1987. The capacity of the Hopfield associative memory. Information Theory, IEEE Transactions on, 33(4), pp.461-482
1987
Earlier work this paper cites.
Kanter, I. and Sompolinsky, H., 1987. Associative recall of memory without errors. Physical Review A, 35(1), p.380
1987
Earlier work this paper cites.

Similar

Baldi, P. and Venkatesh, S.S., 1987. Number of stable points for spin-glasses and neural networks of higher orders. Physical Review Letters, 58(9), p.913
1987
Cited alongside, same era.
Gardner, E., 1987. Multiconnected neural network models. Journal of Physics A: Mathematical and General, 20(11), p.3453
1987
Cited alongside, same era.
Abbott, L.F. and Arian, Y., 1987. Storage capacity of generalized networks. Physical Review A, 36(10), p.5091
1987
Cited alongside, same era.
Horn, D. and Usher, M., 1988. Capacities of multiconnected memory models. Journal de Physique, 49(3), pp.389-395
1988
Cited alongside, same era.
Simard, P.Y., Steinkraus, D. and Platt, J.C., 2003, August. Best practices for convolutional neural networks applied to visual document analysis. In null (p. 958). IEEE
2003
Cited alongside, same era.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M. and Huang, F., 2006. A tutorial on energy-based learning. Predicting structured data, 1, p.0
2006
Cited alongside, same era.
Hinton, G.E., Osindero, S. and Teh, Y.W., 2006. A fast learning algorithm for deep belief nets. Neural computation, 18(7), pp.1527-1554
2006
Cited alongside, same era.

Then

Hinton, G.E. and Salakhutdinov, R.R., 2006. Reducing the dimensionality of data with neural networks. Science, 313(5786), pp.504-507
2006
Later among the works it cites.
Nair, V. and Hinton, G.E., 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) (pp. 807-814)
2010
Later among the works it cites.
Glorot, X., Bordes, A. and Bengio, Y., 2011. Deep sparse rectifier neural networks. In International Conference on Artificial Intelligence and Statistics (pp. 315-323)
2011
Later among the works it cites.
Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. ImageNet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105)
2012
Later among the works it cites.
Wan, L., Zeiler, M., Zhang, S., LeCun, Y. and Fergus, R., 2013. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13) (pp. 1058-1066)
2013
Later among the works it cites.
Goodfellow, I.J., Shlens, J. and Szegedy, C., 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572
Original
2014
Later among the works it cites.
Rasmus, A., Berglund, M., Honkala, M., Valpola, H. and Raiko, T., 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems (pp. 3546-3554)
2015
Later among the works it cites.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…