The advantage function measures how much better or worse it is to take a specific action in a given state than to act according to the current policy on average. Formally it is defined as A(s, a) = Q(s, a) – V(s), the difference between the action-value function and the state-value function.
Reinforcement learning algorithms use the advantage to decide which actions to reinforce. Because it subtracts the state’s baseline value, it isolates the contribution of the action itself, which makes policy gradient updates far less noisy than updates based on raw returns. Actor-critic methods, A2C and A3C, proximal policy optimization and many value-based architectures depend on it.
How the advantage function is defined
The advantage function compares two expectations under the same policy pi. The Q-function Q(s, a) gives the expected discounted return after taking action a in state s and then following pi. The value function V(s) gives the expected return when the action itself is also drawn from pi. Their difference is the advantage.
Two properties follow directly from the definition. First, the expected advantage under the policy is zero: averaging A(s, a) over actions drawn from pi gives exactly zero. Second, for an optimal policy the advantage of the best action is zero and every other action has a negative advantage.
| Quantity | Notation | Question it answers |
|---|---|---|
| State-value function | V(s) | How good is this state under the policy |
| Action-value function | Q(s, a) | How good is this action in this state |
| Advantage function | A(s, a) = Q(s, a) – V(s) | How much better is this action than average |
| TD error | r + gamma V(s’) – V(s) | One-step sample estimate of the advantage |
The sign carries the practical meaning: a positive advantage says the action outperformed the policy’s typical choice, and a negative advantage says it underperformed.
Why advantages reduce variance in policy gradients
Policy gradient methods increase the log-probability of actions in proportion to a weighting signal. The REINFORCE algorithm uses the full return, which is unbiased but has very high variance, because the return depends on every later action and every random transition.
Subtracting a state-dependent baseline leaves the expected gradient unchanged while reducing its variance. The value function V(s) is close to the variance-minimizing baseline, and subtracting it turns the weighting signal into the advantage. In effect, the agent stops reinforcing actions merely because they occurred in a good state and credits only the improvement the action produced.
The table shows the main weighting signals used in policy gradient estimators and how they trade bias against variance.
| Weighting signal | Bias | Variance | Typical use |
|---|---|---|---|
| Full return | None | Very high | Basic REINFORCE |
| Return minus baseline | None | High | REINFORCE with baseline |
| One-step TD error | High if critic is poor | Low | Simple actor-critic |
| n-step advantage | Moderate | Moderate | A2C and A3C |
| Generalized advantage estimation | Tunable | Tunable | PPO and TRPO |
Estimating the advantage in practice
The true advantage is unknown, so algorithms estimate it from sampled trajectories and a learned critic. The simplest estimator is the one-step temporal difference error, which uses the reward, the discount factor gamma and the critic’s value of the next state.
Generalized advantage estimation (GAE) blends n-step estimators with an exponential weight lambda. With lambda = 0 it reduces to the one-step TD error, which has low variance but inherits the critic’s bias. With lambda = 1 it becomes the Monte Carlo return minus the baseline, which is unbiased but noisy. Common defaults are gamma = 0.99 and lambda = 0.95.
| Setting | Value range | Effect |
|---|---|---|
| Discount factor gamma | 0.9 to 0.999 | Higher values look further ahead and increase variance |
| GAE lambda | 0.9 to 0.99 | Higher values reduce bias and increase variance |
| Rollout length | 128 to 2,048 steps | Longer rollouts give better long-horizon estimates |
| Advantage normalization | Per batch | Stabilizes step sizes across tasks |
Most implementations also normalize advantages per batch to zero mean and unit standard deviation. This keeps the effective learning rate stable even when reward scales differ widely between environments.
Where the advantage function appears
The advantage function is a central signal in modern deep reinforcement learning, used both to train policies and to structure value networks. The table lists the main families that rely on it.
| Algorithm family | Role of the advantage |
|---|---|
| Actor-critic, A2C, A3C | Weights the actor’s policy gradient |
| PPO and TRPO | Scales the clipped or constrained surrogate objective |
| Dueling DQN | Separate network stream combined with a value stream |
| Advantage-weighted regression | Weights supervised updates in offline learning |
| Group relative policy optimization | Replaces the critic with group-normalized rewards |
Group relative policy optimization, used in training large language models, shows that the critic is optional: it estimates the advantage by comparing each sampled answer’s reward with the mean and spread of rewards in its group.
Limitations and common pitfalls
Advantage estimates are only as good as the critic and the sampling scheme behind them. The following issues account for many unstable training runs.
- Biased critic: a poorly fitted value function pushes the policy in the wrong direction, especially with low lambda.
- Wrong episode boundaries: failing to distinguish true terminal states from time-limit truncation corrupts the bootstrapped value.
- Stale advantages: reusing advantages for too many epochs after the policy has changed makes updates off-policy.
- Sparse rewards: when nearly all rewards are zero, advantages carry little signal regardless of the estimator.
- Confusing advantage with reward: a positive reward can still produce a negative advantage if the state normally yields more.
A plain-language summary
Imagine a basketball player who usually scores about 50% of shots from a certain spot. If a new shooting technique scores 60% from that spot, it is better than usual; if it scores 40%, it is worse. The advantage function asks exactly this question for every decision an agent makes.
It does not ask whether the outcome was good in absolute terms. It asks whether the choice beat the usual result in that situation. This makes learning fairer and steadier, because the agent stops taking credit for lucky situations and learns which choices truly helped.
An example from practice
A robotics team trains a simulated robotic arm to stack blocks using proximal policy optimization. Each training batch contains 16 parallel environments with rollouts of 2,048 steps, and the reward combines progress toward the target with a bonus for a completed stack.
Early experiments use raw returns as the weighting signal, and the success rate stalls at around 30% with large swings between runs. After switching to generalized advantage estimation with gamma 0.99 and lambda 0.95, normalizing advantages per batch and fixing the handling of time-limit truncation, training curves become smooth.
The arm reaches a stacking success rate above 85% in simulation with roughly half the environment steps needed before. The change came from a cleaner advantage signal, not from a larger network.