Boltzmann machine

On this page 8

A Boltzmann machine is a stochastic, symmetrically connected neural network of binary units that learns a probability distribution over its inputs by assigning an energy to every possible state. Among neural networks, it is the classic example of an energy-based generative model: states with low energy are probable, and learning reshapes the energy landscape so that observed data becomes likely.

Geoffrey Hinton and Terrence Sejnowski introduced the model in the mid-1980s, extending the deterministic Hopfield network with random sampling borrowed from statistical physics. Its restricted variant later helped revive deep learning through layer-wise pretraining.

How a Boltzmann machine assigns energy to states

A Boltzmann machine defines an energy function over joint states of all its units, and the probability of each state follows the Boltzmann distribution from thermodynamics. For binary units s with symmetric weights w and biases b, the energy is E(s) = -sum(w_ij s_i s_j) - sum(b_i s_i), and the probability of a state is proportional to exp(-E(s)/T), where T is a temperature parameter usually fixed at 1.

Units are split into visible units, which are clamped to data, and hidden units, which capture latent structure. Each unit switches on with a probability given by the logistic sigmoid of its total input, so the network never settles into one answer but samples states. Repeated sampling with Gibbs sampling eventually reaches thermal equilibrium, where state frequencies match the model distribution.

The table summarizes the components that define the model and the role each plays.

ComponentDefinitionRole in the model
Visible unitsBinary units tied to observed dataRepresent inputs such as pixels
Hidden unitsBinary units never observed directlyCapture latent features
Symmetric weightsw_ij equals w_ji, no self-loopsGuarantee a well-defined energy
BiasesPer-unit offset termsSet baseline activation probability
TemperatureScaling factor T in the exponentControls randomness of sampling
Partition functionSum of exp(-E) over all statesNormalizes probabilities, intractable
Core components of a Boltzmann machine

Learning rule and the role of sampling

Training maximizes the log-likelihood of the data, and the gradient for each weight takes a strikingly local form: the difference between two correlations. The update is proportional to <s_i s_j>_data - <s_i s_j>_model, where the first term is measured with visible units clamped to training examples (the positive phase) and the second with the network running freely (the negative phase).

The difficulty lies in the negative phase. Estimating model correlations requires samples from equilibrium, which in a fully connected network can take an impractically long time to reach because of the many interacting units. The partition function is a sum over 2 to the power n states, so exact computation is impossible for more than a few dozen units. Early implementations relied on simulated annealing, gradually lowering the temperature, which made training extremely slow and unstable.

This is the main reason that general Boltzmann machines remained a theoretical model. Practical progress came only when the connectivity was restricted so that sampling could proceed in parallel blocks.

Restricted and deep variants

The most important variant is the restricted Boltzmann machine (RBM), which removes all connections within a layer and keeps only visible-to-hidden connections. This bipartite structure makes hidden units conditionally independent given the visible ones, so an entire layer can be sampled in one step.

In 2002, Hinton proposed contrastive divergence, which approximates the negative phase with only one or a few Gibbs steps starting from the data. Although the gradient is biased, it works well in practice and made RBMs trainable on real datasets. Stacking RBMs layer by layer produced deep belief networks, and joint training of several hidden layers produced deep Boltzmann machines.

VariantConnectivityTraining methodPractical status
General Boltzmann machineAll units connectedAnnealed Gibbs samplingMostly theoretical
Restricted Boltzmann machineBipartite, visible to hidden onlyContrastive divergenceWidely used in the 2000s
Gaussian-Bernoulli RBMBipartite, real-valued visiblesContrastive divergence, careful scalingContinuous data such as audio
Deep belief networkStacked RBMs, top layer undirectedGreedy layer-wise pretrainingHistorical milestone
Deep Boltzmann machineSeveral undirected hidden layersMean-field plus persistent chainsResearch use
Main variants of the Boltzmann machine

A Boltzmann machine is best positioned as the probabilistic successor of the Hopfield network and a predecessor of modern deep generative models. The comparison clarifies what each model optimizes and how it produces outputs.

ModelUnit behaviorObjectiveTypical output
Hopfield networkDeterministic threshold unitsMinimize energy to a stored patternRecalled memory
Boltzmann machineStochastic binary unitsMaximize data likelihoodSamples from learned distribution
AutoencoderDeterministic, feedforwardMinimize reconstruction errorCompressed code and reconstruction
Variational autoencoderDeterministic with sampled latentMaximize evidence lower boundGenerated samples
Diffusion modelDeterministic denoising networkLearn to reverse added noiseHigh-quality generated images
Boltzmann machines and neighboring models

Unlike a feedforward neural network or a multilayer perceptron, a Boltzmann machine has no fixed input-to-output direction; any subset of units can be clamped and the rest inferred. This makes it naturally suited to filling in missing values, but it also makes inference expensive compared with a single forward pass.

Applications and practical guidance

Boltzmann machines, mostly in restricted form, were used for unsupervised feature learning, collaborative filtering and dimensionality reduction. An RBM-based model was a strong component in the Netflix Prize recommendation competition, and stacked RBMs provided pretraining for deep networks before better initialization and ReLU activations made it unnecessary.

SituationRecommendationReason
Teaching energy-based learningSmall RBM on binary dataMechanism is transparent
Modeling binary co-occurrence dataRBM with persistent chainsCaptures joint dependencies
Large-scale image generationPrefer diffusion or autoregressive modelsFar better sample quality
Quantum annealing researchBoltzmann sampling on annealersHardware samples low-energy states
Pretraining deep classifiersAvoid in modern pipelinesStandard initialization suffices
When a Boltzmann machine is a reasonable choice

Hyperparameters matter: learning rates around 0.001 to 0.1, mini-batches of tens to hundreds, weight decay and momentum are typical starting points for RBMs, and monitoring reconstruction error alone is misleading because it does not measure likelihood.

Limitations and common misconceptions

The defining limitation is the intractable partition function, which makes both exact likelihood evaluation and unbiased gradients unavailable for realistic sizes. Several misunderstandings recur in practice.

  • Low reconstruction error is not good likelihood; a model can reconstruct well while sampling poorly.
  • Contrastive divergence is not exact maximum likelihood, and its bias can trap the model in spurious modes.
  • Mixing between modes is slow, so samples may stay near one pattern for many steps.
  • Boltzmann machines are not obsolete in theory; energy-based ideas continue in modern score-based and energy-based models.
  • The name does not imply physical hardware, although some analog and quantum devices implement Boltzmann sampling.

The simple explanation

A Boltzmann machine is a network of tiny on-off switches that influence each other. Instead of giving one fixed answer, it keeps flipping switches at random, but some combinations are much more likely than others.

Training adjusts the connections so that patterns resembling real examples become the most likely combinations. Think of a landscape of hills and valleys: learning digs valleys where the data lives, and a ball rolling at random spends most of its time in those valleys. The key takeaway is that the model learns what typical data looks like and can then produce new examples or fill in missing parts.

How it is used in practice

A music streaming service wants to suggest songs to listeners who have rated only a few tracks. The data team represents each listener as a vector of about 5,000 binary visible units, one per popular track, set to 1 when the listener liked it, and trains a restricted Boltzmann machine with 300 hidden units using contrastive divergence over several days of training on a single server.

For a new listener, the known likes are clamped to the visible units, the hidden units are sampled to infer taste factors such as acoustic jazz or energetic electronic music, and the visible probabilities are then read back. Tracks with high predicted probability but no rating become recommendations.

In offline tests on held-out ratings, the RBM produced more relevant suggestions than simple popularity ranking, and the team kept it as one component of a blended recommender alongside matrix factorization.