Deep reinforcement learning

Also known as: deep RL

On this page 7

Deep reinforcement learning (deep RL) is the branch of reinforcement learning in which deep neural networks approximate the policy, the value function or a model of the environment, allowing an agent to learn from rewards directly from high-dimensional inputs such as images, sensor streams or text. It replaces the lookup tables of classic methods with learned function approximators that generalize across states the agent has never seen.

The approach became prominent when a Deep Q-network learned to play dozens of Atari games from raw pixels, and later when systems combining deep networks, self-play and Monte Carlo tree search reached superhuman level in Go. Today deep RL is used in robotics, resource scheduling, chip placement research and the fine-tuning of large language models with human or automated feedback.

How deep reinforcement learning works

Deep reinforcement learning works by letting an agent interact with an environment framed as a Markov decision process, collecting tuples of state, action, reward and next state, and using them to train a neural network by gradient descent. The network either predicts the expected cumulative reward of actions or directly outputs a probability distribution over actions.

The learning target comes from the Bellman equation or from sampled returns, not from labeled examples. This makes the problem harder than supervised learning: the data distribution changes as the policy changes, targets depend on the network’s own estimates, and rewards may arrive many steps after the decisive action, which is the credit assignment problem.

The table below shows which network components appear in typical deep RL agents and what each one learns.

ComponentNetwork outputUsed by
Value networkState or action value estimateDQN, Double DQN
Policy networkAction probabilities or continuous actionsREINFORCE, PPO, TRPO
Actor and critic pairPolicy plus value baselineA3C, SAC, TD3
World modelPredicted next state and rewardModel-based agents
Reward modelLearned preference scoreFine-tuning language models
Neural network roles in deep RL agents

Main families of deep RL algorithms

Deep RL algorithms fall into value-based, policy-based and actor-critic families, with an orthogonal split between model-free and model-based methods. The choice depends mainly on whether the action space is discrete or continuous and how expensive environment interaction is.

FamilyRepresentative methodsAction spaceKey property
Value-basedDQN, Double DQNDiscreteOff-policy, uses experience replay
Policy gradientREINFORCE, PPODiscrete or continuousDirectly optimizes the policy
Off-policy actor-criticSAC, TD3, DDPGContinuousSample-efficient, reuses old data
Model-basedLearned world models with planningBothFewer real interactions needed
Offline RLConservative value methodsBothLearns from fixed logged data
Major deep RL algorithm families

Proximal policy optimization (PPO) is the most widely used general-purpose method because it is stable and simple to implement, while soft actor-critic (SAC) is a common choice for continuous control in robotics because it reuses data efficiently and encourages exploration through entropy regularization.

Stabilizing training with function approximation

Combining neural networks with bootstrapped value targets is unstable by default, a problem known as the deadly triad of function approximation, bootstrapping and off-policy learning. Practical deep RL relies on a set of engineering techniques that keep training from diverging.

TechniqueProblem addressedHow it helps
Experience replayCorrelated consecutive samplesTrains on shuffled past transitions
Target networkMoving value targetsUpdates target weights slowly
Double estimatorsOverestimated action valuesSeparates action choice and evaluation
Clipped policy updatesDestructive large stepsLimits change per update in PPO
Advantage normalizationHigh-variance gradientsUses baselines such as GAE
Reward scalingExploding value magnitudesKeeps targets in a stable range
Stabilization techniques in deep reinforcement learning

Even with these measures, results are highly sensitive to hyperparameters and random seeds. Credible evaluation therefore reports performance across several seeds with confidence intervals, not a single best run.

Where deep reinforcement learning is applied

Deep reinforcement learning is applied where decisions are sequential, feedback is measurable and a simulator or large log exists. Without one of these conditions, supervised learning or classic optimization is usually the cheaper and safer choice.

  • Games and simulation, where unlimited cheap episodes allow self-play and extensive exploration.
  • Robotic control, usually trained in simulation and transferred to hardware with domain randomization to bridge the sim-to-real gap.
  • Operations and scheduling, such as data center cooling, traffic signal timing and inventory control.
  • Language model alignment, where policy optimization methods such as PPO or group relative policy optimization tune a model against a learned reward model or verifiable rewards.

Limitations and risks of deep RL

The central limitation of deep RL is sample inefficiency: model-free agents often need millions to billions of environment steps, which is affordable in simulation but rarely in the physical world. Other problems follow from the fact that the agent optimizes exactly the reward it is given, not the intention behind it.

Failure modeTypical symptomMitigation
Reward hackingHigh reward, unwanted behaviorCareful reward design, human review
Poor explorationStuck on a weak strategyIntrinsic motivation, entropy bonus
Sparse rewardNo learning signal for long periodsReward shaping, curricula
Overfitting to the simulatorFailure on real hardwareDomain randomization, real fine-tuning
Unsafe explorationDamaging actions during trainingSafe RL constraints, offline pretraining
Common failure modes and mitigations

Deep RL policies are also difficult to interpret and verify, so deployments in safety-relevant settings typically keep a rule-based safety layer or human oversight around the learned controller.

A non-technical view

Think of training a dog with treats. The dog tries things, gets a treat for good behavior and slowly learns what pays off. Deep reinforcement learning teaches a computer program the same way. The program tries actions, receives a score and adjusts itself to earn higher scores over time.

The word deep means the program uses a large neural network as its brain. That network lets it handle complicated inputs such as camera images instead of a short list of numbers. The catch is that it usually needs a huge amount of practice, which is why most training happens in simulations.

A concrete example

A warehouse automation team wants a robotic arm to pick mixed items from bins. Hand-written grasping rules fail on soft packaging and odd shapes, so the engineers build a physics simulation with thousands of randomized objects, lighting conditions and camera positions. They train a soft actor-critic agent that receives a depth image and outputs gripper motions, earning a reward of 1 for each successful lift and a small penalty for collisions.

After about 20 million simulated grasp attempts spread across many parallel simulator instances, the policy succeeds on roughly 90 percent of simulated picks. Transferred to the real arm with domain randomization and a short period of real-world fine-tuning, it reaches a pick success rate in the mid-80s percent on previously unseen items. A conventional safety controller still limits force and workspace, so a wrong decision causes a failed pick rather than damage.