Experience replay

Also known as: replay buffer

On this page 6

Experience replay is a reinforcement learning technique in which an agent stores its past transitions in a memory buffer and repeatedly samples from that memory to update its value function or policy. Each stored transition is typically a tuple of state, action, reward, next state and a termination flag. The memory itself is called a replay buffer, and it is a core component of most off-policy deep reinforcement learning algorithms.

The technique serves two purposes. It breaks the temporal correlation between consecutive experiences, which destabilizes neural network training, and it improves sample efficiency by letting each costly interaction with the environment contribute to many gradient updates instead of one.

How experience replay works

Experience replay works by decoupling data collection from learning. The agent acts in the environment, writes each transition into the buffer and, at regular intervals, draws a random mini-batch of stored transitions to compute a learning update. The batch mixes experiences from many episodes and time steps, so the training data look closer to the independent and identically distributed samples that stochastic gradient descent assumes.

The buffer is usually implemented as a fixed-capacity circular queue. When it is full, the oldest transitions are overwritten first, so the memory holds a sliding window of recent behavior. Learning typically begins only after a warm-up period in which the buffer is filled with a minimum number of transitions, often collected with a random or highly exploratory policy.

The idea dates to early work on connectionist reinforcement learning in the early 1990s, but it became prominent with the Deep Q-network (DQN), where replay combined with a separate target network made it possible to train Q-learning with deep convolutional networks on raw Atari game frames.

The table below lists the key parameters of a replay buffer and the effect each one has on training.

ParameterTypical rangeEffect on learning
Buffer capacity10^4 to 10^6 transitionsLarger buffers add diversity but retain stale data
Mini-batch size32 to 512 transitionsLarger batches reduce gradient noise
Warm-up size1,000 to 50,000 transitionsPrevents overfitting to a nearly empty buffer
Replay ratio0.25 to 8 updates per stepHigher ratios reuse data more aggressively
Sampling strategyUniform or prioritizedDecides which experiences dominate updates
Main replay buffer parameters

Uniform and prioritized replay variants

Experience replay variants differ mainly in how transitions are sampled and in what extra information is stored. The original form samples uniformly, treating every stored transition as equally useful, which is simple and robust but wastes updates on transitions the agent has already learned well.

Prioritized experience replay (PER) samples transitions in proportion to the size of their temporal difference error, so surprising experiences are replayed more often. Because this changes the sampling distribution, PER applies importance-sampling weights to correct the bias, with an exponent that is annealed toward full correction over training. A sum-tree data structure keeps sampling and priority updates at logarithmic cost.

The following table compares the most common variants and when each one is the better choice.

VariantSampling ruleMain advantageTypical use
Uniform replayEqual probability for all transitionsSimple and stableDQN, DDPG, TD3, soft actor-critic
Prioritized replayProportional to TD errorFaster learning from rare eventsRainbow DQN, sparse reward tasks
Hindsight experience replayRelabels goals after the episodeLearning from failed attemptsGoal-conditioned robotics
N-step replayStores multi-step returnsFaster reward propagationRainbow, distributed agents
Distributed replayShared buffer for many actorsHigh data throughputApe-X, R2D2 style systems
Common experience replay variants

Hindsight experience replay deserves special mention for sparse reward problems. After an episode that misses its goal, the agent stores additional copies of the transitions with the goal replaced by a state it actually reached, turning failures into useful training signal.

Experience replay in on-policy and off-policy learning

Experience replay is natural only for off-policy learning, because stored transitions were generated by older versions of the policy. Algorithms such as Q-learning, deep deterministic policy gradient, TD3 and soft actor-critic learn about a target policy from data produced by a different behavior policy, so replayed data remain valid inputs.

On-policy methods such as REINFORCE, A3C and proximal policy optimization assume that training data come from the current policy. They usually discard data after a few epochs instead of keeping a long replay memory. Offline reinforcement learning sits at the opposite extreme: it learns entirely from a fixed dataset, which behaves like a replay buffer that is never refreshed and therefore requires conservative algorithms to avoid overestimating unseen actions.

SettingUse of past dataExample algorithms
Off-policy onlineLong replay buffer, continuous refreshDQN, soft actor-critic, TD3
On-policy onlineShort rollout batch, discarded after usePPO, A3C, REINFORCE
OfflineFixed dataset, no new collectionConservative Q-learning, IQL
Role of stored experience across learning settings

Limitations and practical pitfalls

Experience replay introduces trade-offs that require tuning. Stored data become stale as the policy improves, memory costs grow with buffer size, and reusing the same data too often can lead the network to overfit and lose plasticity.

  • Stale data: very large buffers keep transitions from early, poor policies, which can slow convergence in nonstationary tasks.
  • Memory footprint: storing one million image observations can require tens of gigabytes unless frames are compressed or shared between transitions.
  • Excessive replay ratio: too many updates per environment step can cause overfitting and a loss of network plasticity.
  • Recurrent agents: memory-based policies need stored sequences and burn-in steps for the hidden state, not single transitions.
  • Bias in prioritized replay: omitting importance-sampling correction skews the learned values toward rare, high-error transitions.

A practical rule is to start with uniform replay, a buffer of about 10^5 to 10^6 transitions and a replay ratio near one update per step, then adjust only when learning curves show instability or slow progress.

In plain words

Experience replay is like a student who keeps a notebook of everything that happened during practice and then rereads random pages instead of only remembering the last few minutes. Mixing old and new lessons prevents the student from being thrown off by a streak of similar situations.

For a learning agent, this notebook means each experience can be studied many times. The key takeaway is that the agent learns more from less real-world practice and learns more steadily.

A real-world example

A robotics team trains a robotic arm in simulation to push objects to target positions with soft actor-critic. Each simulated attempt takes several seconds, so collecting data is the main bottleneck. The team stores transitions in a replay buffer of one million entries and performs one gradient update per environment step with mini-batches of 256 transitions.

Rewards are sparse, because the arm earns a reward only when the object lands within 2 centimeters of the target. Early in training, almost every attempt fails. The team adds hindsight experience replay, relabeling each failed episode with the position the object actually reached. Success rates that stayed near zero without relabeling rise above 80 percent after a few hundred thousand steps, an outcome that shows how replay turns failed attempts into training data.