REINFORCE is the classic Monte Carlo policy gradient algorithm in reinforcement learning: it adjusts the parameters of a stochastic policy so that actions followed by high returns become more probable and actions followed by low returns become less probable. Introduced by Ronald J. Williams in 1992, the REINFORCE algorithm is the simplest member of the policy gradient family and the conceptual ancestor of actor-critic methods, proximal policy optimization and group relative policy optimization.
Unlike value-based methods such as Q-learning, REINFORCE optimizes the policy directly without learning an action-value table. It needs only the ability to sample complete episodes and to compute the gradient of the log-probability of chosen actions, which makes it applicable to discrete and continuous action spaces alike.
How REINFORCE estimates the policy gradient
REINFORCE rests on the policy gradient theorem, which states that the gradient of expected return equals the expectation of the return multiplied by the gradient of the log-policy. In code-like notation, the update for one time step is theta += alpha * G_t * grad log pi(a_t | s_t; theta), where G_t is the discounted return from step t to the end of the episode.
The mechanism relies on the likelihood ratio trick, also called the score function estimator. It converts the gradient of an expectation into an expectation of a gradient, so the algorithm never needs a model of the environment dynamics or a differentiable reward. Sampled episodes alone suffice, which is why the method is described as model-free and Monte Carlo.
One training iteration follows a fixed sequence of steps, summarized below.
| Step | Operation | Purpose |
|---|---|---|
| 1. Rollout | Run the current policy to the end of an episode | Collect states, actions and rewards |
| 2. Returns | Compute discounted return for every step | Measure the consequence of each action |
| 3. Baseline | Subtract an estimate of expected return | Reduce gradient variance |
| 4. Gradient | Weight log-probability gradients by advantage | Point toward better behavior |
| 5. Update | Apply gradient ascent with a learning rate | Shift the policy parameters |
Key parameters and design choices
REINFORCE has few hyperparameters, but each one has a strong effect on stability. The learning rate and the discount factor dominate, and the choice of baseline often decides whether training converges at all.
| Setting | Typical range | Effect when too high or too low |
|---|---|---|
| Learning rate | 1e-4 to 1e-2 | High values cause policy collapse; low values stall learning |
| Discount factor | 0.9 to 0.999 | Low values ignore delayed rewards; high values raise variance |
| Episodes per update | 1 to several dozen | More episodes give smoother but costlier gradients |
| Baseline type | None, running mean or value network | No baseline leaves variance very high |
| Entropy bonus | 0 to 0.01 coefficient | Too little causes premature determinism |
The baseline deserves special attention. Subtracting any function of the state that does not depend on the action leaves the gradient unbiased, yet it can shrink variance dramatically. A learned state-value baseline turns the weighting term into an advantage estimate, which is the step that leads conceptually to actor-critic algorithms.
REINFORCE compared with related algorithms
REINFORCE sits at the base of a hierarchy of policy optimization methods. Later algorithms keep its core idea of ascending the log-probability gradient but change how the weighting signal is estimated and how far each update may move the policy.
| Algorithm | Weighting signal | Update constraint | Sample efficiency |
|---|---|---|---|
| REINFORCE | Full Monte Carlo return | None | Low |
| Actor-critic | Bootstrapped TD advantage | None | Moderate |
| TRPO | Advantage estimate | KL divergence trust region | Moderate |
| PPO | Generalized advantage estimation | Clipped probability ratio | Moderate to good |
| GRPO | Return relative to a sampled group | Clipped ratio with KL penalty | Moderate |
The defining difference is the bias-variance trade-off. Full returns make REINFORCE unbiased but noisy, while bootstrapped critics reduce noise at the cost of some bias. Because it is strictly on-policy, REINFORCE also cannot reuse old episodes from a replay buffer without importance sampling corrections.
Where REINFORCE is used today
Despite its age, REINFORCE remains in active use wherever simplicity matters more than sample efficiency or where the reward is a non-differentiable score. It is a standard teaching algorithm, a reliable baseline in research papers and a building block inside larger systems.
- Language model fine-tuning: variants such as REINFORCE with leave-one-out baselines optimize models against reward models or verifiable rewards.
- Neural architecture search: a controller network samples architectures and receives validation accuracy as reward.
- Sequence generation: text or code models are trained on sequence-level metrics that cannot be differentiated.
- Discrete latent variables: the score function estimator trains stochastic components inside larger neural networks.
Limitations and common pitfalls
The central weakness of REINFORCE is high gradient variance, which follows from weighting every action by a single noisy return. Long episodes and sparse rewards worsen the problem, because the credit assignment problem spreads one outcome over many actions.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Erratic reward curve | Unnormalized returns | Normalize returns or add a baseline |
| Policy becomes deterministic early | Entropy collapse | Add an entropy bonus |
| No learning in long episodes | Sparse reward | Reward shaping or shorter horizons |
| Sudden performance crash | Learning rate too large | Lower the rate or clip gradients |
| Biased results after reusing data | Off-policy samples in an on-policy method | Collect fresh episodes per update |
A frequent implementation mistake is weighting every step by the total episode return instead of the reward-to-go. Using only rewards that follow each action is still unbiased and reduces variance noticeably, because earlier rewards cannot be caused by later actions.
In simple terms
Think of learning to throw darts with your eyes half closed. After each round you look at the score. If the round went well, you try to repeat the movements you made; if it went badly, you make those movements less often. REINFORCE teaches a computer program in exactly this way: reward follows the whole attempt, and good attempts are reinforced.
Because a single round can be lucky or unlucky, learning this way is slow and noisy. Comparing each score with an average score helps considerably, since it shows whether a round was truly better than usual.
A practical example
A robotics research group trains a simulated robotic arm to push a block onto a target mark. The policy is a small neural network that outputs a Gaussian distribution over joint torques, and an episode lasts 200 steps with a reward of 1 only when the block reaches the target.
Plain REINFORCE without a baseline learns almost nothing in the first 5,000 episodes, because most episodes return zero. The team adds a small shaped reward for reducing the distance between block and target, a learned state-value baseline, return normalization and an entropy coefficient of 0.005, with batches of 16 episodes per update and a learning rate of 3e-4.
With these changes, the arm reaches a success rate of about 80 percent after roughly 20,000 episodes. The group then uses this REINFORCE result as the baseline against which it measures a PPO implementation, which reaches the same success rate with fewer episodes.