Hopfield network

On this page 7

A Hopfield network is a fully connected recurrent neural network with symmetric weights that stores patterns as minima of an energy function and recalls a complete stored pattern from a partial or corrupted input. John Hopfield described the model in 1982, and it became a foundational idea in neural networks and in artificial intelligence because it framed memory retrieval as a physical relaxation process.

The network works as a content-addressable associative memory: instead of looking up data by address, it is given part of the content and settles into the nearest stored memory. Modern variants with continuous states and exponential capacity are mathematically linked to the attention mechanism used in transformers.

How a Hopfield network stores and recalls patterns

A classical Hopfield network consists of N binary neurons with states of +1 or -1, where every neuron connects to every other neuron and no neuron connects to itself. The weights are symmetric, so the weight from neuron i to neuron j equals the weight from j to i. This symmetry is what guarantees that the network settles instead of oscillating.

Patterns are stored with the Hebbian learning rule: each weight is the sum, over all stored patterns, of the product of the two neurons’ values in that pattern, usually divided by N. Learning is therefore a single pass without gradient descent, which is very different from training with backpropagation.

Recall starts by setting the neurons to a cue pattern. Neurons are then updated, typically asynchronously one at a time, by taking the sign of their weighted input. Each update can only lower or keep the network energy, defined as minus one half of the sum of weight times state products, so the dynamics converge to a stable fixed point called an attractor.

The table below summarizes the main components and their typical settings in the classical model.

ComponentClassical settingPurpose
Neuron statesBinary, +1 or -1Encode one bit of a pattern
Weight matrixSymmetric, zero diagonalGuarantees energy descent
Learning ruleHebbian outer productStores patterns in one pass
Update schemeAsynchronous sign updatesEnsures convergence to a fixed point
Energy functionQuadratic in the statesDefines attractors as memories
Components of the classical Hopfield network

Storage capacity and spurious states

The classical network has a limited storage capacity of roughly 0.138 N random patterns for N neurons before recall breaks down. A network of 1,000 neurons therefore reliably stores only about 138 uncorrelated patterns, and far fewer when the patterns are similar to each other.

Beyond capacity, the energy landscape fills with spurious attractors: mixture states that blend several memories, reversed copies of stored patterns, and spin-glass states that match nothing. Recall then lands in a false memory rather than the intended one.

VariantApproximate capacityMain driver
Classical binary modelAbout 0.138 N patternsCrosstalk between Hebbian terms
Storkey or pseudo-inverse ruleUp to about N patternsReduced interference between memories
Dense associative memoryGrows polynomially with NHigher-order interaction energy
Modern Hopfield networkGrows exponentially with dimensionExponential energy, softmax retrieval
Capacity of Hopfield network variants

Capacity figures assume random, uncorrelated patterns. Real data such as images are strongly correlated, so practical capacity is usually well below the theoretical bound unless patterns are decorrelated first.

The Hopfield network sits between energy-based models and recurrent architectures, and it is easy to confuse with several neighbors. The comparison below shows where it differs in purpose and training.

ModelMain purposeTraining method
Hopfield networkAssociative recall of stored patternsHebbian rule, no gradients
Boltzmann machineProbabilistic generative modelingStochastic sampling with hidden units
Recurrent neural networkProcessing sequences over timeBackpropagation through time
AutoencoderLearning compressed representationsGradient descent on reconstruction
Self-attention layerContext-dependent token mixingEnd-to-end gradient training
Hopfield network and neighboring models

The Boltzmann machine is a stochastic extension that adds hidden units and temperature, which lets it model probability distributions rather than fixed memories. Unlike a sequence model, a Hopfield network has no notion of time steps as input; its recurrence exists only to reach equilibrium.

Modern Hopfield networks and attention

Modern Hopfield networks, introduced around 2016 by Krotov and Hopfield and extended to continuous states in 2020, replace the quadratic energy with sharper functions. The continuous version retrieves a memory in a single update step using a softmax over similarities between the query and the stored patterns.

That update rule is mathematically equivalent to transformer attention: the query is the state pattern, the stored patterns act as keys and values, and the inverse temperature plays the role of the scaling factor in scaled dot-product attention. This connection reframed attention as a form of associative memory retrieval and motivated Hopfield layers for tasks such as multiple-instance learning.

In 2024 John Hopfield shared the Nobel Prize in Physics with Geoffrey Hinton for foundational work enabling machine learning with artificial neural networks, which reflects the model’s role as a bridge between statistical physics and AI.

Practical limitations and design advice

Classical Hopfield networks are rarely used directly in production today, but they remain valuable for teaching, analog hardware research and combinatorial optimization. Practitioners should be aware of the failure modes summarized below.

SymptomLikely causeRemedy
Recall returns a blend of memoriesLoad near capacityStore fewer patterns or use a better rule
Network oscillates between statesSynchronous updatesSwitch to asynchronous updates
Similar patterns are confusedCorrelated memoriesDecorrelate or sparsify the patterns
Optimization finds poor solutionsLocal energy minimaAdd noise or annealing schedules
Common problems and remedies
  • Keep the load well below 0.14 N in the classical model.
  • Use asynchronous updates whenever guaranteed convergence matters.
  • Prefer modern continuous variants when the memory must sit inside a trained deep network.

In everyday language

Think of a landscape with several valleys, where each valley holds one memory. Drop a ball anywhere near a valley and it rolls down to the bottom. A Hopfield network works the same way: give it a blurry or partial picture, and it rolls toward the closest stored memory.

This is why hearing a few notes can bring back a whole song. The key takeaway is that a Hopfield network remembers by content, not by location, but it can only hold a limited number of memories before they start to blur together.

Seen in the real world

A university course on neural networks uses a Hopfield network to demonstrate associative memory. Students encode ten black-and-white images of handwritten digits as 100-pixel binary patterns and store them with the Hebbian rule in a network of 100 neurons.

They then flip 20 percent of the pixels in one digit and present it as a cue. With ten stored patterns, the load is about 0.1 N, and the network restores the original digit within a few update sweeps. When the students increase the load to 25 patterns, recall begins to fail and produces mixed, ghost-like digits.

The exercise shows the core trade-off clearly: the network corrects noise reliably, but only while the number of memories stays below capacity.