Actor-critic is a family of reinforcement learning algorithms that combine two learned components: an actor, which represents the policy and chooses actions, and a critic, which estimates a value function and judges how good those actions were. The critic’s evaluation replaces noisy full-episode returns in the policy update, which makes learning faster and more stable than pure policy gradient methods.
The actor-critic method sits between value-based methods such as Q-learning and policy-based methods such as REINFORCE. It underlies many widely used algorithms, including A3C, PPO, DDPG, TD3 and soft actor-critic.
How the actor and the critic interact
The actor and the critic form a feedback loop in which each step of experience updates both. The actor, a parameterized policy, observes a state and samples an action. The environment returns a reward and the next state. The critic then computes a temporal difference (TD) error, the difference between the observed reward plus the discounted value of the next state and its own prediction for the current state.
That TD error serves two purposes. The critic uses it as a regression signal to improve its value estimate, and the actor uses it as an estimate of the advantage, increasing the probability of actions that turned out better than expected and decreasing it for worse ones. Updates can happen after every step, without waiting for an episode to end.
The table lists the core quantities in a typical actor-critic update.
| Quantity | Role | Learned by |
|---|---|---|
| Policy pi(a given s) | Maps states to action probabilities | Actor |
| State value V(s) | Predicts expected discounted return | Critic |
| TD error delta | Reward plus discounted V(s’) minus V(s) | Computed, not learned |
| Advantage A(s, a) | How much better an action was than average | Estimated from the critic |
| Discount factor gamma | Weights future rewards, typically 0.95 to 0.999 | Set as a hyperparameter |
Why a critic reduces variance
A critic reduces variance because it replaces the full Monte Carlo return with a learned estimate that depends on far fewer random events. In REINFORCE, the gradient is weighted by the sum of all future rewards, which varies widely from episode to episode and forces very small learning rates and many samples.
Subtracting a baseline, the state value, leaves the expected gradient unchanged while lowering its variance. Bootstrapping from the critic’s own prediction lowers variance further but introduces bias, because the critic is never perfectly accurate. Generalized advantage estimation (GAE) controls this trade-off with a parameter lambda, typically between 0.9 and 0.97, which blends one-step TD errors with longer returns.
The following table compares common ways to weight the policy gradient.
| Signal | Bias | Variance |
|---|---|---|
| Full return | None | Very high |
| Return minus baseline | None | High |
| One-step TD error | Highest | Lowest |
| n-step advantage | Moderate | Moderate |
| GAE with lambda | Tunable | Tunable |
Main actor-critic variants
Actor-critic variants differ mainly in whether they learn on-policy or off-policy, whether actions are discrete or continuous, and how they constrain policy updates. The choice among them is usually driven by the action space and by how expensive environment interaction is.
| Algorithm | Policy type | Distinctive feature |
|---|---|---|
| A2C and A3C | On-policy, stochastic | Parallel workers collect diverse experience |
| PPO | On-policy, stochastic | Clipped objective limits each update |
| DDPG | Off-policy, deterministic | Critic learns Q(s, a) for continuous actions |
| TD3 | Off-policy, deterministic | Twin critics and delayed actor updates |
| Soft actor-critic | Off-policy, stochastic | Maximizes reward plus policy entropy |
Off-policy variants reuse past transitions through experience replay, which makes them far more sample efficient in robotics and simulation where each step is costly. On-policy variants such as PPO are generally simpler to tune and more robust, which explains their popularity in game playing and in fine-tuning language models from human feedback.
Comparison with value-based and policy gradient methods
Actor-critic methods combine the strengths of both main families of reinforcement learning. Value-based methods such as Q-learning and DQN derive the policy implicitly by taking the action with the highest estimated value, which is efficient for discrete actions but awkward for continuous control. Pure policy gradient methods optimize the policy directly and handle any action space, but they learn slowly because of high gradient variance.
| Aspect | Value-based | Policy gradient | Actor-critic |
|---|---|---|---|
| Explicit policy | No | Yes | Yes |
| Continuous actions | Difficult | Natural | Natural |
| Gradient variance | Not applicable | High | Reduced |
| Stochastic policies | Limited | Yes | Yes |
| Stability risks | Overestimation | Slow convergence | Coupled learning dynamics |
Practical pitfalls and tuning
The central difficulty is that the actor and critic learn from each other’s moving targets. A poor critic sends misleading gradients to the actor, and a rapidly changing actor makes the critic’s value targets drift. Many training failures come from this coupled instability rather than from the environment itself.
Common remedies include target networks that update slowly, gradient clipping, separate optimizers for actor and critic, and running the critic for several updates per actor update. Monitoring the critic’s explained variance alongside episode return helps reveal whether a plateau comes from an inaccurate critic or from a policy that has stopped exploring.
- Critic learning rate too low: the actor follows stale value estimates and stalls.
- Overestimated Q-values: deterministic variants exploit critic errors, which TD3 counters with twin critics.
- Premature policy collapse: an entropy bonus or the maximum-entropy objective of soft actor-critic keeps exploration alive.
- Unnormalized advantages and rewards: scaling them often decides whether training converges at all.
- Shared network trunks: sharing layers saves computation but can let value loss overwhelm policy features.
The idea in brief
Picture a learner driver and an instructor in the passenger seat. The driver makes the moves, and the instructor says after each one whether it went better or worse than expected. The driver adjusts from that feedback instead of waiting until the end of the lesson.
In actor-critic learning, the actor is the driver and the critic is the instructor. Both improve together: the instructor gets better at judging, and the driver gets better at acting. The key point is that quick, step-by-step feedback makes learning steadier than judging only by the final result.
A practical scenario
A warehouse automation team trains a robotic arm in simulation to pick irregular parcels from a bin. The action space is continuous, covering joint velocities and gripper force, and each simulated grasp takes about half a second, so sample efficiency matters.
The team chooses soft actor-critic with two critic networks and a replay buffer of one million transitions. The reward combines a bonus for a stable lift with small penalties for collisions and excessive force. After roughly two million simulation steps, the policy reaches a grasp success rate above 90 percent in simulation, and domain randomization of friction and lighting lets it transfer to the physical arm with modest fine-tuning. The entropy term kept the arm trying varied grasps early on, which prevented it from settling on a fragile single strategy.