Deep Q-network

Also known as: DQN · deep Q-learning

On this page 7

A Deep Q-network (DQN) is a reinforcement learning method that uses a deep neural network to approximate the Q-function, the expected cumulative reward of taking an action in a given state and acting well afterward. It extends classic Q-learning, which stores values in a table, to problems with huge or continuous state spaces such as raw screen pixels, and it is often called deep Q-learning.

DQN became a landmark in artificial intelligence when one network with fixed hyperparameters learned to play dozens of Atari 2600 games directly from pixels, many at or above human level. Its two stabilizing ideas, experience replay and a target network, remain standard components of value-based deep reinforcement learning.

How a Deep Q-network learns action values

A Deep Q-network learns by repeatedly adjusting its weights so that its predicted value Q(s, a) moves toward a temporal difference target derived from the Bellman equation. For each transition (s, a, r, s’), the target is r + gamma max Q(s’, a’), where gamma is the discount factor, typically between 0.95 and 0.99.

The network takes a state as input and outputs one Q-value per discrete action in a single forward pass. The agent then selects the action with the highest value, while an epsilon-greedy policy occasionally picks a random action to keep exploring. The loss is the squared or Huber loss between prediction and target, minimized with stochastic gradient descent.

The core components of a DQN agent and their roles are summarized below.

ComponentRoleTypical form
Online networkPredicts Q-values used for actingConvolutional or fully connected network
Target networkSupplies stable bootstrap targetsPeriodic copy of the online network
Replay bufferStores past transitions for reuseCircular buffer of up to 1 million transitions
Exploration policyBalances trying and exploiting actionsEpsilon-greedy with decaying epsilon
Loss functionMeasures temporal difference errorHuber loss
Core components of a DQN agent

Why experience replay and target networks are needed

Experience replay and target networks are needed because naive Q-learning with a neural network is unstable and often diverges. The combination of function approximation, bootstrapping and off-policy updates is known as the deadly triad, and DQN was the first widely successful design to tame it in practice.

Experience replay stores transitions in a buffer and trains on random minibatches, often of 32 transitions. Random sampling breaks the strong correlation between consecutive frames and lets each experience be reused many times, which improves data efficiency.

The target network is a frozen copy of the online network used only to compute targets. Without it, every update shifts the target it is chasing. Copying the weights every thousands of steps, or blending them slowly with a soft update, keeps the regression target stable long enough for learning to converge.

The table below lists the main hyperparameters and their typical ranges in published DQN setups.

HyperparameterTypical rangeEffect when poorly set
Discount factor0.95 to 0.99Too low ignores long-term reward
Replay buffer size100,000 to 1,000,000 transitionsToo small overfits recent experience
Target update interval1,000 to 10,000 stepsToo short destabilizes targets
Final epsilon0.01 to 0.1Too high wastes actions on random play
Learning rate0.00001 to 0.001Too high causes divergence
Key DQN hyperparameters

Variants and improvements of DQN

The original DQN has been extended by several improvements that each fix a specific weakness. The best known is Double DQN, which reduces the systematic overestimation bias caused by using the same network to select and evaluate the maximizing action.

The table below compares the main variants and the problem each one addresses.

VariantProblem addressedKey change
Double DQNOverestimated Q-valuesOnline network selects, target network evaluates
Dueling DQNPoor value estimates for similar actionsSeparate value and advantage streams
Prioritized replayUniform sampling wastes updatesSamples high-error transitions more often
Distributional DQNOnly the mean return is modeledLearns the full return distribution
Noisy networksCrude epsilon-greedy explorationLearned noise in network weights
RainbowFragmented improvementsCombines six extensions in one agent
Main extensions of the Deep Q-network

Combining these ideas in Rainbow produced substantially better Atari performance than any single extension, which shows that the improvements are largely complementary rather than overlapping.

DQN compared with policy-based methods

DQN is a value-based, off-policy, model-free method, while algorithms such as REINFORCE, proximal policy optimization and actor-critic methods learn a policy directly. The choice between them depends mostly on the action space and on how expensive data collection is.

PropertyDeep Q-networkPolicy gradient methods
Action spaceDiscrete onlyDiscrete or continuous
Data reuseHigh through replay bufferLow for on-policy methods
Policy typeDeterministic greedy plus explorationStochastic by design
Training stabilitySensitive to hyperparametersOften smoother with clipping
Value-based and policy-based approaches

For continuous control, the max over actions in the DQN target becomes intractable, so practitioners use related actor-critic methods such as deep deterministic policy gradient, TD3 or soft actor-critic, which keep the Q-function idea but add a separate policy network.

Limitations and common pitfalls

DQN has well-known limitations that practitioners must plan for. It is sample-hungry, often needing tens of millions of frames to master a single Atari game, and it struggles with sparse rewards because epsilon-greedy exploration rarely stumbles onto distant goals.

  • Reward scale matters: unclipped or poorly normalized rewards can produce exploding gradients.
  • Partial observability breaks the Markov assumption, which is why Atari agents stack the last four frames as input.
  • Overestimation persists in vanilla DQN, so Double DQN is a sensible default.
  • Evaluation noise is high, so results should be averaged over several random seeds.

Finally, a trained DQN reflects its training environment closely. Small changes in dynamics or visuals can degrade performance sharply, so deployment outside simulation usually requires domain randomization or further fine-tuning.

A plain-language summary

A Deep Q-network is a program that learns to make good choices by trial and error. For every situation, it estimates how much future reward each possible move will bring and then picks the most promising one.

It learns like a person reviewing old game recordings. Instead of learning only from the latest moment, it keeps a memory of past experiences and studies random pieces of it again and again. The key takeaway is that DQN turns raw observations into a score for each action, and better scores lead to better decisions.

An example from practice

A logistics company wants a warehouse robot simulator to learn which of 6 movement actions to take on a grid with shelves, moving obstacles and pick-up points. The state is an 84 x 84 overhead image of the robot’s surroundings, which is far too large for a Q-table.

Engineers train a Double DQN with a 500,000-transition replay buffer, a target network refreshed every 8,000 steps and epsilon decaying from 1.0 to 0.05 over the first two million steps. After about 20 million simulated steps, the agent completes picking routes roughly a quarter faster than the rule-based baseline in simulation. The team then uses randomized shelf layouts during further training so that the policy transfers to new warehouse configurations.