Deep deterministic policy gradient

Also known as: DDPG

On this page 7

Deep deterministic policy gradient (DDPG) is a model-free, off-policy actor-critic algorithm that learns a deterministic policy for continuous action spaces by training a neural network actor along the gradient of a learned Q-function. In reinforcement learning, it extends the ideas of the Deep Q-network to problems where actions are real-valued vectors, such as joint torques, steering angles or throttle levels.

DDPG was introduced in 2015 and builds directly on the deterministic policy gradient theorem. It combines four ingredients: an actor network that outputs actions, a critic network that estimates action values, an experience replay buffer and slowly updated target networks. The algorithm made deep continuous control practical, and its weaknesses shaped successors such as TD3 and Soft actor-critic.

How DDPG trains the actor and the critic

DDPG alternates between collecting experience and updating two networks. The agent acts with the current actor plus exploration noise, stores each transition of state, action, reward and next state in a replay buffer, and then samples random mini-batches from that buffer for learning.

The critic learns the action-value function by minimizing the temporal difference error against a Bellman target: the reward plus the discounted value of the next state, where the next action comes from the target actor. The actor is updated by the deterministic policy gradient, which follows the critic’s gradient with respect to the action and backpropagates it through the actor’s parameters.

The table below lists the components and the function each one serves.

ComponentRoleWhy it is needed
Actor networkMaps a state to one continuous actionAvoids maximizing Q over infinite actions
Critic networkEstimates Q for a state-action pairSupplies the gradient that improves the actor
Replay bufferStores past transitions for reuseBreaks sample correlation, enables off-policy learning
Target networksSlow copies of actor and criticStabilize the bootstrapped Bellman targets
Exploration noisePerturbs actions during data collectionCompensates for a deterministic policy
Core components of DDPG

The target networks are updated by soft updates (Polyak averaging), typically with a coefficient tau around 0.001 to 0.005, so they trail the learned networks smoothly instead of being copied periodically.

Key hyperparameters and their effects

DDPG is notoriously sensitive to hyperparameters, so understanding each setting matters more than for many on-policy methods. Small changes in learning rates or noise scale can decide whether training converges or collapses.

The following table gives commonly used ranges for standard continuous control benchmarks.

HyperparameterTypical rangeWhat it affects
Actor learning rate1e-4 to 1e-3Speed and stability of policy updates
Critic learning rate1e-3 to 3e-4Accuracy of value estimates
Discount factor0.98 to 0.99Horizon of credited rewards
Soft update tau0.001 to 0.005Lag of the target networks
Replay buffer size1e5 to 1e6 transitionsDiversity and age of training data
Exploration noise sigma0.1 to 0.2 of action rangeBreadth of exploration
Typical DDPG hyperparameters

The original paper used Ornstein-Uhlenbeck noise, a temporally correlated process suited to physical systems with inertia. Later studies showed that uncorrelated Gaussian noise usually works just as well and is simpler to tune.

DDPG sits between value-based methods such as the Deep Q-network and stochastic policy gradient methods such as Proximal policy optimization. Its off-policy nature gives good sample efficiency, while its deterministic actor makes exploration and stability harder.

The comparison below places DDPG among the algorithms most often considered for the same tasks.

AlgorithmAction spacePolicy typeMain strength
Deep Q-networkDiscreteImplicit, greedy on QSimple and stable for discrete actions
DDPGContinuousDeterministic actorSample-efficient off-policy learning
TD3ContinuousDeterministic actorFixes overestimation with twin critics
Soft actor-criticContinuousStochastic, entropy-regularizedRobust exploration and stability
Proximal policy optimizationBothStochastic, on-policyReliable, easy to tune, less sample-efficient
DDPG and neighboring algorithms

In current practice, TD3 and Soft actor-critic usually replace DDPG in new projects because they reach similar or better returns with far less tuning. DDPG remains valuable as a conceptual baseline and as the simplest member of this family.

Where DDPG fits in practice

DDPG suits problems with continuous, low to moderate dimensional actions and an environment where collecting data is expensive enough that reusing transitions matters. Typical settings include robotic manipulation in simulation, locomotion benchmarks, process control and energy management.

The decision table below helps judge when DDPG is a reasonable choice and when another method fits better.

SituationRecommended choiceReason
Discrete actions onlyDeep Q-network or Double DQNNo continuous gradient is needed
Continuous actions, limited tuning timeSoft actor-critic or TD3More stable defaults
Teaching or quick baselineDDPGSmallest working actor-critic design
Fixed dataset, no interactionOffline reinforcement learning methodsPlain DDPG overestimates unseen actions
Very fast simulator, many parallel workersProximal policy optimizationSample efficiency matters less
Choosing DDPG or an alternative

Before training, normalizing observations and scaling actions to the range from -1 to 1 with a tanh output layer are close to mandatory. Reward scaling also matters, because very large rewards inflate critic errors.

Failure modes and common mistakes

The central weakness of DDPG is Q-value overestimation. The actor actively seeks actions where the critic is optimistic, and errors in the critic therefore grow and feed back into the policy, which leads to brittle learning curves and sudden performance collapses.

  • Overestimation bias: the actor exploits critic errors; twin critics in TD3 address this.
  • High seed variance: identical settings can give very different results, so evaluations need several random seeds.
  • Insufficient exploration: too little noise leaves the policy stuck in a poor local optimum.
  • Unscaled inputs and rewards: large magnitudes destabilize the critic.
  • Evaluating with noise on: performance must be measured with the deterministic policy alone.

Treating a single good training run as proof of success is a frequent mistake. Reporting averages over five or more seeds with confidence intervals gives a realistic picture of DDPG performance.

Stripped to the essentials

Think of learning to drive with a coach in the passenger seat. You decide exactly how far to turn the wheel and press the pedal, and the coach scores each move. Over time, you adjust your moves in the direction the coach rates higher.

DDPG works the same way with two neural networks. One, the actor, chooses a precise action. The other, the critic, judges how good that action is. The actor keeps nudging its choices toward what the critic prefers, and both learn from a memory of past experiences.

Where it is applied

A robotics research group trains a simulated seven-joint robotic arm to push objects to target positions on a table. Each action is a vector of seven continuous joint velocity commands between -1 and 1, so a discrete method such as a Deep Q-network would require an impractical number of action combinations.

The group uses DDPG with a replay buffer of one million transitions, actor and critic learning rates of 1e-4 and 1e-3, tau of 0.005 and Gaussian exploration noise with sigma 0.1. After about two million environment steps, the arm reaches the goal in roughly 80 percent of evaluation episodes, averaged over five seeds.

Two of the five seeds show performance drops late in training, a typical sign of critic overestimation. The group then switches to TD3 with the same network sizes and obtains a more stable success rate near 90 percent, while keeping DDPG as the reference baseline in its report.