TD3

Also known as: Twin delayed deep deterministic policy gradient

On this page 7

TD3, short for twin delayed deep deterministic policy gradient, is an off-policy actor-critic algorithm for reinforcement learning in continuous action spaces. It extends deep deterministic policy gradient (DDPG) with three targeted changes that reduce overestimation bias in the learned Q-function and make training markedly more stable.

Introduced in 2018 by Scott Fujimoto, Herke van Hoof and David Meger, TD3 became a standard baseline for robotic and simulated control tasks, alongside soft actor-critic. Its design shows how errors in value estimation propagate into a deterministic policy and how small structural fixes can control them.

How TD3 learns a deterministic policy

TD3 learns a deterministic policy (the actor) that maps each state directly to one continuous action, together with two Q-function critics that estimate the expected discounted return of a state-action pair. The actor improves by following the gradient of the critic with respect to the action, as in DDPG.

Training is off-policy: the agent acts in the environment with added exploration noise, stores transitions in an experience replay buffer, and updates its networks from random minibatches of past data. Slowly moving target networks for the actor and both critics provide stable regression targets for the Bellman update.

One TD3 training iteration follows a fixed order of steps.

  1. Select an action from the actor, add Gaussian exploration noise, step the environment and store the transition.
  2. Sample a minibatch and compute the target action from the target actor plus clipped smoothing noise.
  3. Compute the target value as the reward plus the discounted minimum of the two target critics.
  4. Update both critics by minimizing the squared temporal difference error against that shared target.
  5. Every second iteration, update the actor and softly update all target networks.

The three changes that define TD3

TD3 is defined by three modifications to DDPG, each addressing a specific failure mode of deterministic actor-critic learning. The name refers to the first two: twin critics and delayed updates.

MechanismWhat it doesProblem it addresses
Clipped double Q-learningUses the minimum of two critics in the targetOverestimation bias of a single critic
Delayed policy updatesUpdates the actor less often than the criticsActor chasing a noisy, unsettled critic
Target policy smoothingAdds clipped noise to the target actionExploitation of sharp peaks in the critic
The three core mechanisms of TD3

Clipped double Q-learning is the most important change. Function approximation errors make some Q-values too high, and a policy that maximizes Q actively seeks out those errors. Taking the minimum of two independently initialized critics yields a deliberately pessimistic target, which trades a small underestimation for far less harmful overestimation.

Delayed policy updates let the value estimate settle before the policy moves, reducing the variance that feeds back into the critic. Target policy smoothing acts as a regularizer: by averaging the target over nearby actions, it assumes that similar actions should have similar values, so the actor cannot exploit narrow spikes in the learned Q-surface.

Key hyperparameters and typical values

TD3 is known for working with relatively few tuned hyperparameters, and the values from the original paper remain a reasonable starting point for most continuous control benchmarks.

HyperparameterTypical valueEffect when changed
Policy delay d2Higher values slow actor learning but stabilize it
Target noise std0.2 (scaled to action range)More smoothing, less sharp exploitation
Noise clip c0.5Limits how far smoothed actions can move
Exploration noise std0.1Controls breadth of data collection
Soft update rate tau0.005Lower values give slower, steadier targets
Discount factor gamma0.99Sets the effective planning horizon
Replay buffer size1 million transitionsLarger buffers keep more diverse data
Learning rateAbout 3e-4 for actor and criticsToo high causes divergence
Common TD3 hyperparameters

Implementations often begin with a warm-up period of random actions, commonly 10,000 to 25,000 steps, so that the replay buffer contains varied data before learning starts. Actions are usually bounded with a tanh output and rescaled to the environment limits.

TD3 compared with DDPG and soft actor-critic

TD3 sits between DDPG, its direct predecessor, and soft actor-critic (SAC), a contemporary method that also uses twin critics but learns a stochastic policy with an entropy bonus. The comparison below highlights the practical differences.

PropertyDDPGTD3SAC
Policy typeDeterministicDeterministicStochastic
Number of criticsOneTwo, minimum usedTwo, minimum used
ExplorationExternal noiseExternal Gaussian noiseEntropy maximization
Overestimation controlNoneStrongStrong
Hyperparameter sensitivityHighModerateLow to moderate
Typical sample efficiencyModerateHighHigh
TD3 versus related continuous control algorithms

Compared with on-policy methods such as proximal policy optimization, TD3 reuses past experience and is therefore far more sample-efficient, which matters when environment interaction is slow or expensive. The price is greater sensitivity to replay data quality and less straightforward support for discrete actions, where TD3 does not apply directly.

Limitations and practical pitfalls

TD3 is robust by deep reinforcement learning standards, but it shares the brittleness of off-policy actor-critic methods and has a few specific weaknesses.

  • Exploration is undirected: fixed Gaussian noise struggles in environments with sparse reward, where SAC or intrinsic motivation methods often do better.
  • Underestimation can slow learning in some tasks, because the minimum of two critics is systematically pessimistic.
  • Results vary across random seeds, so fair evaluation requires several seeds and reporting of variance.
  • Unnormalized observations and unscaled rewards are a common cause of divergence in practice.
  • Offline use is unsafe without changes; TD3+BC adds a behavior cloning term to keep the policy close to the dataset.

Because TD3 produces a single deterministic action per state, it is well suited to deployment on controllers that need repeatable behavior, but it gives no built-in measure of action uncertainty.

In simple terms

TD3 helps a computer learn smooth, precise movements. Think of how hard to push a joint or how far to turn a wheel. The computer tries things and gets a score. It keeps two independent judges that guess how good each move is. It then trusts the more cautious judge. Overly hopeful guesses would lead learning astray.

It also lets the judges settle down first. Only then does it change its behavior. It ignores oddly high scores for one exact move. The result is steadier learning than the older method it improved.

A practical example

A robotics research group trains a simulated seven-joint robot arm to push objects to target positions before transferring the controller to hardware. Each action is a vector of seven joint torques, so a continuous control method is required, and every hour of simulation is costly on their shared compute cluster.

The team starts with DDPG, but its learning curves collapse after about 300,000 steps as the critic overestimates certain torque combinations. Switching to TD3 with a policy delay of 2, target noise of 0.2 and a 1 million transition buffer, they observe stable improvement across five random seeds.

After roughly one million environment steps, the TD3 policy reaches the target position in about 85% of test episodes, and its deterministic output makes the controller easier to validate on the physical arm.