Actor-critic

Also known as: actor–critic method

On this page 7

Actor-critic is a family of reinforcement learning algorithms that combine two learned components: an actor, which represents the policy and chooses actions, and a critic, which estimates a value function and judges how good those actions were. The critic’s evaluation replaces noisy full-episode returns in the policy update, which makes learning faster and more stable than pure policy gradient methods.

The actor-critic method sits between value-based methods such as Q-learning and policy-based methods such as REINFORCE. It underlies many widely used algorithms, including A3C, PPO, DDPG, TD3 and soft actor-critic.

How the actor and the critic interact

The actor and the critic form a feedback loop in which each step of experience updates both. The actor, a parameterized policy, observes a state and samples an action. The environment returns a reward and the next state. The critic then computes a temporal difference (TD) error, the difference between the observed reward plus the discounted value of the next state and its own prediction for the current state.

That TD error serves two purposes. The critic uses it as a regression signal to improve its value estimate, and the actor uses it as an estimate of the advantage, increasing the probability of actions that turned out better than expected and decreasing it for worse ones. Updates can happen after every step, without waiting for an episode to end.

The table lists the core quantities in a typical actor-critic update.

QuantityRoleLearned by
Policy pi(a given s)Maps states to action probabilitiesActor
State value V(s)Predicts expected discounted returnCritic
TD error deltaReward plus discounted V(s’) minus V(s)Computed, not learned
Advantage A(s, a)How much better an action was than averageEstimated from the critic
Discount factor gammaWeights future rewards, typically 0.95 to 0.999Set as a hyperparameter
Core quantities in one actor-critic update

Why a critic reduces variance

A critic reduces variance because it replaces the full Monte Carlo return with a learned estimate that depends on far fewer random events. In REINFORCE, the gradient is weighted by the sum of all future rewards, which varies widely from episode to episode and forces very small learning rates and many samples.

Subtracting a baseline, the state value, leaves the expected gradient unchanged while lowering its variance. Bootstrapping from the critic’s own prediction lowers variance further but introduces bias, because the critic is never perfectly accurate. Generalized advantage estimation (GAE) controls this trade-off with a parameter lambda, typically between 0.9 and 0.97, which blends one-step TD errors with longer returns.

The following table compares common ways to weight the policy gradient.

SignalBiasVariance
Full returnNoneVery high
Return minus baselineNoneHigh
One-step TD errorHighestLowest
n-step advantageModerateModerate
GAE with lambdaTunableTunable
Gradient weighting signals and their trade-offs

Main actor-critic variants

Actor-critic variants differ mainly in whether they learn on-policy or off-policy, whether actions are discrete or continuous, and how they constrain policy updates. The choice among them is usually driven by the action space and by how expensive environment interaction is.

AlgorithmPolicy typeDistinctive feature
A2C and A3COn-policy, stochasticParallel workers collect diverse experience
PPOOn-policy, stochasticClipped objective limits each update
DDPGOff-policy, deterministicCritic learns Q(s, a) for continuous actions
TD3Off-policy, deterministicTwin critics and delayed actor updates
Soft actor-criticOff-policy, stochasticMaximizes reward plus policy entropy
Widely used actor-critic algorithms

Off-policy variants reuse past transitions through experience replay, which makes them far more sample efficient in robotics and simulation where each step is costly. On-policy variants such as PPO are generally simpler to tune and more robust, which explains their popularity in game playing and in fine-tuning language models from human feedback.

Comparison with value-based and policy gradient methods

Actor-critic methods combine the strengths of both main families of reinforcement learning. Value-based methods such as Q-learning and DQN derive the policy implicitly by taking the action with the highest estimated value, which is efficient for discrete actions but awkward for continuous control. Pure policy gradient methods optimize the policy directly and handle any action space, but they learn slowly because of high gradient variance.

AspectValue-basedPolicy gradientActor-critic
Explicit policyNoYesYes
Continuous actionsDifficultNaturalNatural
Gradient varianceNot applicableHighReduced
Stochastic policiesLimitedYesYes
Stability risksOverestimationSlow convergenceCoupled learning dynamics
Actor-critic compared with neighboring approaches

Practical pitfalls and tuning

The central difficulty is that the actor and critic learn from each other’s moving targets. A poor critic sends misleading gradients to the actor, and a rapidly changing actor makes the critic’s value targets drift. Many training failures come from this coupled instability rather than from the environment itself.

Common remedies include target networks that update slowly, gradient clipping, separate optimizers for actor and critic, and running the critic for several updates per actor update. Monitoring the critic’s explained variance alongside episode return helps reveal whether a plateau comes from an inaccurate critic or from a policy that has stopped exploring.

  • Critic learning rate too low: the actor follows stale value estimates and stalls.
  • Overestimated Q-values: deterministic variants exploit critic errors, which TD3 counters with twin critics.
  • Premature policy collapse: an entropy bonus or the maximum-entropy objective of soft actor-critic keeps exploration alive.
  • Unnormalized advantages and rewards: scaling them often decides whether training converges at all.
  • Shared network trunks: sharing layers saves computation but can let value loss overwhelm policy features.

The idea in brief

Picture a learner driver and an instructor in the passenger seat. The driver makes the moves, and the instructor says after each one whether it went better or worse than expected. The driver adjusts from that feedback instead of waiting until the end of the lesson.

In actor-critic learning, the actor is the driver and the critic is the instructor. Both improve together: the instructor gets better at judging, and the driver gets better at acting. The key point is that quick, step-by-step feedback makes learning steadier than judging only by the final result.

A practical scenario

A warehouse automation team trains a robotic arm in simulation to pick irregular parcels from a bin. The action space is continuous, covering joint velocities and gripper force, and each simulated grasp takes about half a second, so sample efficiency matters.

The team chooses soft actor-critic with two critic networks and a replay buffer of one million transitions. The reward combines a bonus for a stable lift with small penalties for collisions and excessive force. After roughly two million simulation steps, the policy reaches a grasp success rate above 90 percent in simulation, and domain randomization of friction and lighting lets it transfer to the physical arm with modest fine-tuning. The entropy term kept the arm trying varied grasps early on, which prevented it from settling on a fragile single strategy.