Double DQN

On this page 7

Double DQN is a value-based algorithm in reinforcement learning that extends the Deep Q-network (DQN) by decoupling action selection from action evaluation in the learning target. This single change reduces the systematic overestimation of Q-values that standard DQN suffers from, and it usually yields more stable training and better policies.

The method was introduced by Hado van Hasselt, Arthur Guez and David Silver in 2015, building on the earlier Double Q-learning idea for tabular agents. Within artificial intelligence, it became a standard component of later value-based agents, including the Rainbow combination of DQN improvements.

Why standard DQN overestimates action values

Standard DQN overestimates values because its target uses the same noisy estimates both to pick the best next action and to score it. The DQN target is the reward plus the discounted maximum Q-value of the next state, taken from the target network. When the estimates contain random errors, the maximum tends to select actions whose values happen to be overestimated by noise.

This effect is called maximization bias. Even if each individual estimate is unbiased, the expected maximum of noisy estimates is higher than the maximum of the true values. Through bootstrapping in the Bellman equation, the inflated targets feed back into later updates, so the error can accumulate across states and distort the learned policy.

SourceMechanismTypical effect
Function approximation errorNetwork outputs deviate from true valuesRandom errors that the max operator favors
Environment stochasticityNoisy rewards and transitionsLucky samples inflate estimates
Many actionsThe max is taken over more noisy valuesBias grows with the size of the action space
BootstrappingTargets reuse current estimatesBias propagates to earlier states
Sources of overestimation in Q-learning targets

How the Double DQN target is computed

Double DQN computes its target by letting the online network choose the action and the target network evaluate it. For a transition with reward r and next state s’, the online network finds the action with the highest Q-value in s’. The target network then supplies the value of that action, which is discounted and added to r.

Written compactly, the target is y = r + gamma * Q_target(s', argmax_a Q_online(s', a)), while the standard DQN target is y = r + gamma * max_a Q_target(s', a). Because the two networks have partly independent errors, an action that one network overrates is often not overrated by the other, so the upward bias shrinks.

StepDQNDouble DQN
Select next actionTarget network, greedy maxOnline network, greedy argmax
Evaluate selected actionTarget networkTarget network
Extra networks neededNone beyond the target networkNone beyond the target network
Bias tendencyUpward overestimationMuch smaller, occasionally downward
Target computation in DQN and Double DQN

The modification costs almost nothing. Double DQN reuses the existing target network of DQN instead of training a second independent estimator, so the only extra work is one additional forward pass of the online network on the next state.

Training setup and key hyperparameters

Double DQN keeps the training pipeline of DQN unchanged: an experience replay buffer, epsilon-greedy exploration, a periodically synchronized target network and minibatch gradient descent on the temporal difference error. Typical values depend on the environment, but Atari-style settings give a useful reference point.

HyperparameterTypical rangeWhat it affects
Discount factor0.95 to 0.99Planning horizon and variance of targets
Replay buffer size100,000 to 1,000,000 transitionsData diversity and memory use
Target network updateEvery 1,000 to 30,000 steps, or soft updatesStability versus speed of learning
Final epsilon0.01 to 0.1Residual exploration after annealing
Minibatch size32 to 256Gradient noise and throughput
Common hyperparameter ranges

The target network update interval matters more in Double DQN than in DQN. If the target network is synchronized too often, the two networks become nearly identical and the decoupling benefit disappears; if it is synchronized too rarely, learning slows down because targets lag behind.

Double DQN addresses one specific weakness of DQN, and it combines well with other fixes that target different weaknesses. Several neighboring methods in reinforcement learning apply the same double-estimator principle to other settings.

MethodProblem addressedRelation to Double DQN
Double Q-learningMaximization bias in tabular Q-learningOriginal idea with two independent tables
Dueling DQNInefficient value learning across actionsDifferent architecture, often combined
Prioritized replayUniform sampling wastes updatesOrthogonal improvement, often combined
RainbowSeveral DQN weaknesses at onceIncludes the Double DQN target
TD3Overestimation in continuous controlUses the minimum of two critics
Related methods and the problem each addresses

Limitations and common misconceptions

Double DQN reduces but does not eliminate estimation bias. Because the online and target networks share history and are periodically synchronized, their errors are correlated, so some overestimation remains. In certain environments the method can even underestimate values slightly, which may slow learning where optimistic estimates would have helped exploration.

  • It is not a new architecture: the network design stays the same; only the target formula changes.
  • It does not fix exploration: sparse reward problems still need better exploration strategies.
  • It applies to discrete actions: continuous control uses related ideas such as clipped double Q-learning in TD3.
  • Gains vary by task: improvements are large where overestimation is severe and small where it is mild.

The short version

A learning agent constantly guesses how good each possible move is. If it always trusts its single most optimistic guess, it keeps fooling itself, because the guess that looks best is often the one with the biggest lucky error.

Double DQN asks two slightly different versions of the agent instead of one. The first picks the move that looks best, and the second gives an independent opinion on how good that move really is. Getting a second opinion keeps the agent honest, so it learns more realistic expectations and makes better decisions.

Where it makes a difference

In the original evaluation on dozens of Atari 2600 games, standard DQN produced value estimates that climbed far above the returns its policy actually achieved, and in some games the inflated values coincided with collapses in score. Double DQN kept its estimates much closer to the true returns and reached higher median scores across the benchmark.

A similar pattern appears in applied settings. A team training an agent to control heating and cooling in an office building, with around a dozen discrete setpoint actions and noisy occupancy patterns, might find that a plain DQN agent repeatedly favors extreme setpoints whose value it overrates. Switching to the Double DQN target, with no other change, typically produces steadier value curves and a policy that saves energy without breaking comfort limits.