Experience replay is a reinforcement learning technique in which an agent stores its past transitions in a memory buffer and repeatedly samples from that memory to update its value function or policy. Each stored transition is typically a tuple of state, action, reward, next state and a termination flag. The memory itself is called a replay buffer, and it is a core component of most off-policy deep reinforcement learning algorithms.
The technique serves two purposes. It breaks the temporal correlation between consecutive experiences, which destabilizes neural network training, and it improves sample efficiency by letting each costly interaction with the environment contribute to many gradient updates instead of one.
How experience replay works
Experience replay works by decoupling data collection from learning. The agent acts in the environment, writes each transition into the buffer and, at regular intervals, draws a random mini-batch of stored transitions to compute a learning update. The batch mixes experiences from many episodes and time steps, so the training data look closer to the independent and identically distributed samples that stochastic gradient descent assumes.
The buffer is usually implemented as a fixed-capacity circular queue. When it is full, the oldest transitions are overwritten first, so the memory holds a sliding window of recent behavior. Learning typically begins only after a warm-up period in which the buffer is filled with a minimum number of transitions, often collected with a random or highly exploratory policy.
The idea dates to early work on connectionist reinforcement learning in the early 1990s, but it became prominent with the Deep Q-network (DQN), where replay combined with a separate target network made it possible to train Q-learning with deep convolutional networks on raw Atari game frames.
The table below lists the key parameters of a replay buffer and the effect each one has on training.
| Parameter | Typical range | Effect on learning |
|---|---|---|
| Buffer capacity | 10^4 to 10^6 transitions | Larger buffers add diversity but retain stale data |
| Mini-batch size | 32 to 512 transitions | Larger batches reduce gradient noise |
| Warm-up size | 1,000 to 50,000 transitions | Prevents overfitting to a nearly empty buffer |
| Replay ratio | 0.25 to 8 updates per step | Higher ratios reuse data more aggressively |
| Sampling strategy | Uniform or prioritized | Decides which experiences dominate updates |
Uniform and prioritized replay variants
Experience replay variants differ mainly in how transitions are sampled and in what extra information is stored. The original form samples uniformly, treating every stored transition as equally useful, which is simple and robust but wastes updates on transitions the agent has already learned well.
Prioritized experience replay (PER) samples transitions in proportion to the size of their temporal difference error, so surprising experiences are replayed more often. Because this changes the sampling distribution, PER applies importance-sampling weights to correct the bias, with an exponent that is annealed toward full correction over training. A sum-tree data structure keeps sampling and priority updates at logarithmic cost.
The following table compares the most common variants and when each one is the better choice.
| Variant | Sampling rule | Main advantage | Typical use |
|---|---|---|---|
| Uniform replay | Equal probability for all transitions | Simple and stable | DQN, DDPG, TD3, soft actor-critic |
| Prioritized replay | Proportional to TD error | Faster learning from rare events | Rainbow DQN, sparse reward tasks |
| Hindsight experience replay | Relabels goals after the episode | Learning from failed attempts | Goal-conditioned robotics |
| N-step replay | Stores multi-step returns | Faster reward propagation | Rainbow, distributed agents |
| Distributed replay | Shared buffer for many actors | High data throughput | Ape-X, R2D2 style systems |
Hindsight experience replay deserves special mention for sparse reward problems. After an episode that misses its goal, the agent stores additional copies of the transitions with the goal replaced by a state it actually reached, turning failures into useful training signal.
Experience replay in on-policy and off-policy learning
Experience replay is natural only for off-policy learning, because stored transitions were generated by older versions of the policy. Algorithms such as Q-learning, deep deterministic policy gradient, TD3 and soft actor-critic learn about a target policy from data produced by a different behavior policy, so replayed data remain valid inputs.
On-policy methods such as REINFORCE, A3C and proximal policy optimization assume that training data come from the current policy. They usually discard data after a few epochs instead of keeping a long replay memory. Offline reinforcement learning sits at the opposite extreme: it learns entirely from a fixed dataset, which behaves like a replay buffer that is never refreshed and therefore requires conservative algorithms to avoid overestimating unseen actions.
| Setting | Use of past data | Example algorithms |
|---|---|---|
| Off-policy online | Long replay buffer, continuous refresh | DQN, soft actor-critic, TD3 |
| On-policy online | Short rollout batch, discarded after use | PPO, A3C, REINFORCE |
| Offline | Fixed dataset, no new collection | Conservative Q-learning, IQL |
Limitations and practical pitfalls
Experience replay introduces trade-offs that require tuning. Stored data become stale as the policy improves, memory costs grow with buffer size, and reusing the same data too often can lead the network to overfit and lose plasticity.
- Stale data: very large buffers keep transitions from early, poor policies, which can slow convergence in nonstationary tasks.
- Memory footprint: storing one million image observations can require tens of gigabytes unless frames are compressed or shared between transitions.
- Excessive replay ratio: too many updates per environment step can cause overfitting and a loss of network plasticity.
- Recurrent agents: memory-based policies need stored sequences and burn-in steps for the hidden state, not single transitions.
- Bias in prioritized replay: omitting importance-sampling correction skews the learned values toward rare, high-error transitions.
A practical rule is to start with uniform replay, a buffer of about 10^5 to 10^6 transitions and a replay ratio near one update per step, then adjust only when learning curves show instability or slow progress.
In plain words
Experience replay is like a student who keeps a notebook of everything that happened during practice and then rereads random pages instead of only remembering the last few minutes. Mixing old and new lessons prevents the student from being thrown off by a streak of similar situations.
For a learning agent, this notebook means each experience can be studied many times. The key takeaway is that the agent learns more from less real-world practice and learns more steadily.
A real-world example
A robotics team trains a robotic arm in simulation to push objects to target positions with soft actor-critic. Each simulated attempt takes several seconds, so collecting data is the main bottleneck. The team stores transitions in a replay buffer of one million entries and performs one gradient update per environment step with mini-batches of 256 transitions.
Rewards are sparse, because the arm earns a reward only when the object lands within 2 centimeters of the target. Early in training, almost every attempt fails. The team adds hindsight experience replay, relabeling each failed episode with the position the object actually reached. Success rates that stayed near zero without relabeling rise above 80 percent after a few hundred thousand steps, an outcome that shows how replay turns failed attempts into training data.