Policy · reinforcement learning

Also known as: RL policy

On this page 7

In reinforcement learning, a policy is the mapping from states to actions that defines how an agent behaves: given what the agent observes, the policy specifies which action to take or, more generally, a probability distribution over actions. Written as π(a | s), the RL policy is the object that every reinforcement learning algorithm ultimately tries to improve.

The goal of the agent is to find an optimal policy, one that maximizes the expected discounted cumulative reward from every state. Value functions, models of the environment and replay buffers are all supporting tools; the policy is what gets deployed when training ends.

How a policy defines agent behavior

A policy turns the Markov decision process into a concrete stream of behavior. At each time step the agent observes a state s, the policy selects an action a, and the environment returns a reward and a next state. Repeating this loop generates trajectories, and the quality of a policy is measured by the expected return those trajectories collect.

Every policy induces its own value function V^π(s) and action-value function Q^π(s, a), which estimate the return obtained by following that policy. The policy improvement theorem states that acting greedily with respect to Q^π yields a policy at least as good as π, which is the engine behind policy iteration and many later methods.

The table below lists the standard ways a policy can be specified and where each form appears.

Policy formDefinitionTypical setting
Deterministic policya = μ(s), one action per stateDDPG, TD3, greedy control
Stochastic policyπ(a | s), a distribution over actionsREINFORCE, PPO, soft actor-critic
Tabular policyLookup table indexed by discrete stateSmall grid worlds, planning
Parameterized policyπ_θ(a | s) with weights θ, often a neural networkDeep reinforcement learning
Implicit policyDerived from a value estimate, such as argmax QQ-learning, deep Q-network
Main forms of a reinforcement learning policy

Deterministic and stochastic policies compared

Whether a policy should be deterministic or stochastic depends on the environment and the training method. In a fully observed Markov decision process, an optimal deterministic policy always exists, yet stochastic policies are often easier to learn and safer to explore with.

AspectDeterministic policyStochastic policy
ExplorationNeeds added noise or epsilon-greedyBuilt in through sampling
Partial observabilityCan loop or get stuckRandomness can break ambiguity
Gradient estimationDeterministic policy gradient via QLikelihood-ratio policy gradient
Multi-agent gamesExploitable by opponentsCan represent mixed strategies
DeploymentRepeatable, easy to auditOften switched to the mean action
Trade-offs between deterministic and stochastic policies

For discrete action spaces, stochastic policies usually output a softmax distribution over actions. For continuous action spaces, they commonly output the mean and standard deviation of a Gaussian distribution, sometimes squashed through tanh to respect action bounds, as in soft actor-critic.

How policies are learned

Algorithms differ mainly in whether they represent the policy explicitly and how they update it. Value-based methods learn Q and derive the policy implicitly, while policy-based methods adjust policy parameters directly by gradient ascent on expected return.

Method familyHow the policy is obtainedExamples
Dynamic programmingAlternate evaluation and greedy improvement using a known modelPolicy iteration, value iteration
Value-basedGreedy or epsilon-greedy over learned Q-valuesQ-learning, SARSA, Double DQN
Policy gradientGradient ascent on log-probability weighted by returnREINFORCE
Actor-criticActor is the policy, critic estimates value to cut varianceA3C, PPO, soft actor-critic
Imitation and inverse RLPolicy fitted to expert demonstrationsBehavior cloning, inverse reinforcement learning
Families of methods and how they obtain the policy

Modern actor-critic methods add safeguards against destructive updates. Trust region policy optimization limits the KL divergence between old and new policies, and proximal policy optimization clips the probability ratio, typically to the range 0.8 to 1.2. Entropy regularization keeps the policy from collapsing too early onto a single action.

Behavior policy, target policy and data use

Many algorithms distinguish the behavior policy that collects data from the target policy being evaluated or improved. This distinction separates on-policy from off-policy learning and determines how data can be reused.

  • On-policy learning: behavior and target policies are the same, as in SARSA and PPO, so old data becomes stale after each update.
  • Off-policy learning: the target policy learns from data produced by another policy, as in Q-learning, enabling experience replay.
  • Importance sampling reweights returns by the ratio of target to behavior probabilities, at the cost of higher variance.
  • Offline reinforcement learning learns a policy from a fixed dataset and must avoid actions poorly covered by that data.

In large language model fine-tuning, the language model itself acts as the policy: each generated token is an action, and methods such as PPO or group relative policy optimization update it against a reward signal while a KL penalty to a reference model limits drift.

Common pitfalls when designing a policy

Policy learning fails in characteristic ways, and recognizing the symptoms speeds up debugging. The table maps frequent problems to their usual causes and remedies.

SymptomLikely causeTypical fix
Premature convergenceEntropy collapses earlyAdd entropy bonus or more exploration
Reward hackingPolicy exploits a flawed reward functionRedesign reward, add constraints
Performance collapse after updateStep size too largeClip ratio, lower learning rate
Good in simulation, poor in realitySim-to-real gapDomain randomization, real data
High variance in returnsNoisy gradient estimatesBaselines, advantage estimation
Policy failure modes and fixes

A further misconception is that a policy must be optimal to be useful. In practice, robustness, safety constraints and sample efficiency often matter more than squeezing out the last fraction of return, which is why safe reinforcement learning treats constraint satisfaction as part of the policy objective.

The simple explanation

A policy is the decision rule an agent follows. Think of it as a playbook: for every situation the agent might face, the playbook says what to do next, or how likely each move should be.

At first the playbook is poor, close to random guessing. Each time the agent acts and receives a reward or a penalty, the learning algorithm edits the playbook so that moves leading to good results become more likely. Training is the process of improving the policy, and the finished policy is what the agent actually uses.

How it is used in practice

A warehouse robotics team trains a robotic arm to pick mixed items from bins. The state combines a depth camera image and joint angles, and the action is a continuous vector of seven joint velocities. The team uses a neural network policy trained with soft actor-critic in simulation, outputting a Gaussian distribution over velocities.

During training, the stochastic policy explores different grasp angles, guided by a reward of plus one for a successful lift and small penalties for collisions. After about 2 million simulated steps with randomized lighting, friction and object shapes, the policy reaches roughly 90 percent success in simulation.

For deployment, the team switches to the deterministic mean action for repeatable behavior and adds a safety layer that rejects commands exceeding torque limits. On the real cell, the policy achieves a pick success rate in the mid 80s, and targeted fine-tuning on real grasps closes most of the remaining gap.