REINFORCE

Also known as: REINFORCE algorithm

On this page 7

REINFORCE is the classic Monte Carlo policy gradient algorithm in reinforcement learning: it adjusts the parameters of a stochastic policy so that actions followed by high returns become more probable and actions followed by low returns become less probable. Introduced by Ronald J. Williams in 1992, the REINFORCE algorithm is the simplest member of the policy gradient family and the conceptual ancestor of actor-critic methods, proximal policy optimization and group relative policy optimization.

Unlike value-based methods such as Q-learning, REINFORCE optimizes the policy directly without learning an action-value table. It needs only the ability to sample complete episodes and to compute the gradient of the log-probability of chosen actions, which makes it applicable to discrete and continuous action spaces alike.

How REINFORCE estimates the policy gradient

REINFORCE rests on the policy gradient theorem, which states that the gradient of expected return equals the expectation of the return multiplied by the gradient of the log-policy. In code-like notation, the update for one time step is theta += alpha * G_t * grad log pi(a_t | s_t; theta), where G_t is the discounted return from step t to the end of the episode.

The mechanism relies on the likelihood ratio trick, also called the score function estimator. It converts the gradient of an expectation into an expectation of a gradient, so the algorithm never needs a model of the environment dynamics or a differentiable reward. Sampled episodes alone suffice, which is why the method is described as model-free and Monte Carlo.

One training iteration follows a fixed sequence of steps, summarized below.

StepOperationPurpose
1. RolloutRun the current policy to the end of an episodeCollect states, actions and rewards
2. ReturnsCompute discounted return for every stepMeasure the consequence of each action
3. BaselineSubtract an estimate of expected returnReduce gradient variance
4. GradientWeight log-probability gradients by advantagePoint toward better behavior
5. UpdateApply gradient ascent with a learning rateShift the policy parameters
Steps of one REINFORCE iteration

Key parameters and design choices

REINFORCE has few hyperparameters, but each one has a strong effect on stability. The learning rate and the discount factor dominate, and the choice of baseline often decides whether training converges at all.

SettingTypical rangeEffect when too high or too low
Learning rate1e-4 to 1e-2High values cause policy collapse; low values stall learning
Discount factor0.9 to 0.999Low values ignore delayed rewards; high values raise variance
Episodes per update1 to several dozenMore episodes give smoother but costlier gradients
Baseline typeNone, running mean or value networkNo baseline leaves variance very high
Entropy bonus0 to 0.01 coefficientToo little causes premature determinism
Main settings and their effects

The baseline deserves special attention. Subtracting any function of the state that does not depend on the action leaves the gradient unbiased, yet it can shrink variance dramatically. A learned state-value baseline turns the weighting term into an advantage estimate, which is the step that leads conceptually to actor-critic algorithms.

REINFORCE sits at the base of a hierarchy of policy optimization methods. Later algorithms keep its core idea of ascending the log-probability gradient but change how the weighting signal is estimated and how far each update may move the policy.

AlgorithmWeighting signalUpdate constraintSample efficiency
REINFORCEFull Monte Carlo returnNoneLow
Actor-criticBootstrapped TD advantageNoneModerate
TRPOAdvantage estimateKL divergence trust regionModerate
PPOGeneralized advantage estimationClipped probability ratioModerate to good
GRPOReturn relative to a sampled groupClipped ratio with KL penaltyModerate
Position of REINFORCE among neighboring methods

The defining difference is the bias-variance trade-off. Full returns make REINFORCE unbiased but noisy, while bootstrapped critics reduce noise at the cost of some bias. Because it is strictly on-policy, REINFORCE also cannot reuse old episodes from a replay buffer without importance sampling corrections.

Where REINFORCE is used today

Despite its age, REINFORCE remains in active use wherever simplicity matters more than sample efficiency or where the reward is a non-differentiable score. It is a standard teaching algorithm, a reliable baseline in research papers and a building block inside larger systems.

  • Language model fine-tuning: variants such as REINFORCE with leave-one-out baselines optimize models against reward models or verifiable rewards.
  • Neural architecture search: a controller network samples architectures and receives validation accuracy as reward.
  • Sequence generation: text or code models are trained on sequence-level metrics that cannot be differentiated.
  • Discrete latent variables: the score function estimator trains stochastic components inside larger neural networks.

Limitations and common pitfalls

The central weakness of REINFORCE is high gradient variance, which follows from weighting every action by a single noisy return. Long episodes and sparse rewards worsen the problem, because the credit assignment problem spreads one outcome over many actions.

SymptomLikely causeRemedy
Erratic reward curveUnnormalized returnsNormalize returns or add a baseline
Policy becomes deterministic earlyEntropy collapseAdd an entropy bonus
No learning in long episodesSparse rewardReward shaping or shorter horizons
Sudden performance crashLearning rate too largeLower the rate or clip gradients
Biased results after reusing dataOff-policy samples in an on-policy methodCollect fresh episodes per update
Symptoms, causes and remedies

A frequent implementation mistake is weighting every step by the total episode return instead of the reward-to-go. Using only rewards that follow each action is still unbiased and reduces variance noticeably, because earlier rewards cannot be caused by later actions.

In simple terms

Think of learning to throw darts with your eyes half closed. After each round you look at the score. If the round went well, you try to repeat the movements you made; if it went badly, you make those movements less often. REINFORCE teaches a computer program in exactly this way: reward follows the whole attempt, and good attempts are reinforced.

Because a single round can be lucky or unlucky, learning this way is slow and noisy. Comparing each score with an average score helps considerably, since it shows whether a round was truly better than usual.

A practical example

A robotics research group trains a simulated robotic arm to push a block onto a target mark. The policy is a small neural network that outputs a Gaussian distribution over joint torques, and an episode lasts 200 steps with a reward of 1 only when the block reaches the target.

Plain REINFORCE without a baseline learns almost nothing in the first 5,000 episodes, because most episodes return zero. The team adds a small shaped reward for reducing the distance between block and target, a learned state-value baseline, return normalization and an entropy coefficient of 0.005, with batches of 16 episodes per update and a learning rate of 3e-4.

With these changes, the arm reaches a success rate of about 80 percent after roughly 20,000 episodes. The group then uses this REINFORCE result as the baseline against which it measures a PPO implementation, which reaches the same success rate with fewer episodes.