Cumulative reward

Also known as: expected return

On this page 7

Cumulative reward is the sum of all rewards an agent receives over a sequence of interactions with its environment, usually weighted by a discount factor so that near-term rewards count more than distant ones. In reinforcement learning, a core area of artificial intelligence, maximizing the expected value of this sum is the formal objective of every agent, which is why the quantity is also called the return or, in expectation, the expected return.

A single reward says only how good one step was. The cumulative reward says how good an entire course of action was, which allows an agent to accept short-term costs for long-term gains.

How cumulative reward is defined

The cumulative reward from time step t, written G_t, adds every future reward the agent receives from that point onward. In the general discounted form, G_t = R_(t+1) + gamma R_(t+2) + gamma^2 R_(t+3) + …, where gamma is the discount factor between 0 and 1. The reward at each step comes from the reward function of the Markov decision process.

The return satisfies a recursive relationship: G_t = R_(t+1) + gamma G_(t+1). This one-line identity is the foundation of the Bellman equation and of every bootstrapping method, because it lets an agent estimate the long-term value of a state from one observed reward plus an estimate of what follows.

Different task structures call for different formulations of the return, as the following table shows.

FormulationFormula sketchTypical setting
Finite-horizon undiscountedSum of rewards up to step TShort episodic tasks with a fixed end
Infinite-horizon discountedSum of gamma^k times R_(t+k+1)Most continuing and episodic tasks
Average rewardLimit of mean reward per stepLong-running control and queuing
Episodic with terminal stateDiscounted sum until terminationGames, robotics trials, puzzles
Formulations of cumulative reward

The role of the discount factor

The discount factor gamma determines how strongly the cumulative reward values the future. It serves two purposes: it keeps the infinite sum mathematically finite when rewards are bounded, and it encodes a preference for earlier rewards or uncertainty about the far future.

A useful rule of thumb is the effective horizon of about 1 / (1 – gamma) steps. Rewards beyond that horizon contribute little to the return, so the choice of gamma silently defines how far ahead the agent plans.

GammaEffective horizonAgent behavior
01 stepFully myopic, maximizes only the next reward
0.9About 10 stepsShort-term planning, faster learning
0.99About 100 stepsCommon default in deep reinforcement learning
0.999About 1,000 stepsLong-horizon tasks, high-variance estimates
1UnboundedValid only for episodic tasks that always end
Discount factor values and their effect

Higher gamma values make the agent more far-sighted but increase the variance of return estimates and slow convergence. Many practitioners treat gamma as a hyperparameter rather than a fixed property of the problem, even though changing it changes what counts as optimal.

Cumulative reward is the raw quantity from which value functions are built, and confusing it with them is a common source of errors. The return is a random variable observed on one trajectory, whereas value functions are its expectations under a policy.

QuantityDefinitionConditioned on
RewardImmediate scalar feedback R_(t+1)One transition
Return G_tDiscounted sum of future rewardsOne sampled trajectory
State value V(s)Expected return from state sState and policy
Action value Q(s, a)Expected return after action a in sState, action and policy
Advantage A(s, a)Q(s, a) minus V(s)Relative merit of an action
Return and its expected forms

The optimal policy is the one that maximizes the expected return from every state. Algorithms differ mainly in how they estimate that expectation: Monte Carlo methods average complete observed returns, while temporal difference learning combines one reward with a bootstrapped estimate of the remainder.

Estimating the return in practice

Agents rarely know the true expected return, so they estimate it from experience, trading bias against variance. Full Monte Carlo returns are unbiased but noisy, because they accumulate the randomness of every future step. One-step temporal difference targets are low-variance but biased by the current value estimate.

  • n-step returns sum n real rewards and then bootstrap, interpolating between the two extremes.
  • Lambda-returns average all n-step returns with exponentially decaying weights, as in TD(lambda).
  • Generalized advantage estimation applies the same idea to advantages and is standard in proximal policy optimization.
  • Reward normalization rescales rewards or returns to stabilize neural network training without changing the optimal policy when done consistently.

In policy gradient methods such as REINFORCE, the sampled return directly scales the gradient of each action probability. Subtracting a baseline, usually the state value, reduces variance without introducing bias.

Pitfalls when maximizing cumulative reward

Because the agent optimizes exactly the cumulative reward it is given, any flaw in the reward design becomes a flaw in behavior. The most serious failures are not algorithmic but come from a mismatch between the return and the designer’s real intent.

ProblemWhat happensTypical remedy
Reward hackingAgent finds loopholes that inflate the returnTighter reward design, constraints, human review
Sparse rewardReturn is almost always zeroReward shaping, intrinsic motivation, curricula
Credit assignmentUnclear which actions earned a late rewardValue functions, eligibility traces
Positive reward loopsAgent circles to collect repeatable rewardsPotential-based shaping, terminal conditions
Poorly chosen gammaAgent ignores delayed consequencesIncrease gamma or use average reward
Common problems and remedies

Another subtlety is that maximizing the expected return ignores risk. Two policies with equal expected return may have very different spreads of outcomes, which is why risk-sensitive and distributional methods model the full distribution of the return rather than its mean alone.

In simple terms

Think of cumulative reward as a running score for a whole game, not for a single move. A chess player who sacrifices a piece loses points on that move but may win the game, and the game result is what counts.

A learning agent works the same way. It adds up all the rewards it expects to collect, usually counting near rewards more than distant ones, much like money today is worth a little more than the same money next year. The agent then chooses actions that make this total as large as possible.

A practical example

A warehouse operator trains a reinforcement learning agent to control a mobile picking robot. Each step costs a reward of -0.1 to encourage speed, a successfully delivered order earns +10, and a collision costs -50. With a discount factor of 0.99, the effective horizon of about 100 steps covers a typical delivery route.

Early versions of the agent learned to wait near the packing station, because idling produced a less negative cumulative reward than risking collisions. The engineers raised the delivery reward and added a small shaping bonus for progress toward the target shelf. After retraining, the average return per hour of simulated operation increased and completed orders rose by roughly a third, while collisions stayed rare, showing how directly the design of the return determines the behavior an agent learns.