Cumulative reward is the sum of all rewards an agent receives over a sequence of interactions with its environment, usually weighted by a discount factor so that near-term rewards count more than distant ones. In reinforcement learning, a core area of artificial intelligence, maximizing the expected value of this sum is the formal objective of every agent, which is why the quantity is also called the return or, in expectation, the expected return.
A single reward says only how good one step was. The cumulative reward says how good an entire course of action was, which allows an agent to accept short-term costs for long-term gains.
How cumulative reward is defined
The cumulative reward from time step t, written G_t, adds every future reward the agent receives from that point onward. In the general discounted form, G_t = R_(t+1) + gamma R_(t+2) + gamma^2 R_(t+3) + …, where gamma is the discount factor between 0 and 1. The reward at each step comes from the reward function of the Markov decision process.
The return satisfies a recursive relationship: G_t = R_(t+1) + gamma G_(t+1). This one-line identity is the foundation of the Bellman equation and of every bootstrapping method, because it lets an agent estimate the long-term value of a state from one observed reward plus an estimate of what follows.
Different task structures call for different formulations of the return, as the following table shows.
| Formulation | Formula sketch | Typical setting |
|---|---|---|
| Finite-horizon undiscounted | Sum of rewards up to step T | Short episodic tasks with a fixed end |
| Infinite-horizon discounted | Sum of gamma^k times R_(t+k+1) | Most continuing and episodic tasks |
| Average reward | Limit of mean reward per step | Long-running control and queuing |
| Episodic with terminal state | Discounted sum until termination | Games, robotics trials, puzzles |
The role of the discount factor
The discount factor gamma determines how strongly the cumulative reward values the future. It serves two purposes: it keeps the infinite sum mathematically finite when rewards are bounded, and it encodes a preference for earlier rewards or uncertainty about the far future.
A useful rule of thumb is the effective horizon of about 1 / (1 – gamma) steps. Rewards beyond that horizon contribute little to the return, so the choice of gamma silently defines how far ahead the agent plans.
| Gamma | Effective horizon | Agent behavior |
|---|---|---|
| 0 | 1 step | Fully myopic, maximizes only the next reward |
| 0.9 | About 10 steps | Short-term planning, faster learning |
| 0.99 | About 100 steps | Common default in deep reinforcement learning |
| 0.999 | About 1,000 steps | Long-horizon tasks, high-variance estimates |
| 1 | Unbounded | Valid only for episodic tasks that always end |
Higher gamma values make the agent more far-sighted but increase the variance of return estimates and slow convergence. Many practitioners treat gamma as a hyperparameter rather than a fixed property of the problem, even though changing it changes what counts as optimal.
Cumulative reward compared with related quantities
Cumulative reward is the raw quantity from which value functions are built, and confusing it with them is a common source of errors. The return is a random variable observed on one trajectory, whereas value functions are its expectations under a policy.
| Quantity | Definition | Conditioned on |
|---|---|---|
| Reward | Immediate scalar feedback R_(t+1) | One transition |
| Return G_t | Discounted sum of future rewards | One sampled trajectory |
| State value V(s) | Expected return from state s | State and policy |
| Action value Q(s, a) | Expected return after action a in s | State, action and policy |
| Advantage A(s, a) | Q(s, a) minus V(s) | Relative merit of an action |
The optimal policy is the one that maximizes the expected return from every state. Algorithms differ mainly in how they estimate that expectation: Monte Carlo methods average complete observed returns, while temporal difference learning combines one reward with a bootstrapped estimate of the remainder.
Estimating the return in practice
Agents rarely know the true expected return, so they estimate it from experience, trading bias against variance. Full Monte Carlo returns are unbiased but noisy, because they accumulate the randomness of every future step. One-step temporal difference targets are low-variance but biased by the current value estimate.
- n-step returns sum n real rewards and then bootstrap, interpolating between the two extremes.
- Lambda-returns average all n-step returns with exponentially decaying weights, as in TD(lambda).
- Generalized advantage estimation applies the same idea to advantages and is standard in proximal policy optimization.
- Reward normalization rescales rewards or returns to stabilize neural network training without changing the optimal policy when done consistently.
In policy gradient methods such as REINFORCE, the sampled return directly scales the gradient of each action probability. Subtracting a baseline, usually the state value, reduces variance without introducing bias.
Pitfalls when maximizing cumulative reward
Because the agent optimizes exactly the cumulative reward it is given, any flaw in the reward design becomes a flaw in behavior. The most serious failures are not algorithmic but come from a mismatch between the return and the designer’s real intent.
| Problem | What happens | Typical remedy |
|---|---|---|
| Reward hacking | Agent finds loopholes that inflate the return | Tighter reward design, constraints, human review |
| Sparse reward | Return is almost always zero | Reward shaping, intrinsic motivation, curricula |
| Credit assignment | Unclear which actions earned a late reward | Value functions, eligibility traces |
| Positive reward loops | Agent circles to collect repeatable rewards | Potential-based shaping, terminal conditions |
| Poorly chosen gamma | Agent ignores delayed consequences | Increase gamma or use average reward |
Another subtlety is that maximizing the expected return ignores risk. Two policies with equal expected return may have very different spreads of outcomes, which is why risk-sensitive and distributional methods model the full distribution of the return rather than its mean alone.
In simple terms
Think of cumulative reward as a running score for a whole game, not for a single move. A chess player who sacrifices a piece loses points on that move but may win the game, and the game result is what counts.
A learning agent works the same way. It adds up all the rewards it expects to collect, usually counting near rewards more than distant ones, much like money today is worth a little more than the same money next year. The agent then chooses actions that make this total as large as possible.
A practical example
A warehouse operator trains a reinforcement learning agent to control a mobile picking robot. Each step costs a reward of -0.1 to encourage speed, a successfully delivered order earns +10, and a collision costs -50. With a discount factor of 0.99, the effective horizon of about 100 steps covers a typical delivery route.
Early versions of the agent learned to wait near the packing station, because idling produced a less negative cumulative reward than risking collisions. The engineers raised the delivery reward and added a small shaping bonus for progress toward the target shelf. After retraining, the average return per hour of simulated operation increased and completed orders rose by roughly a third, while collisions stayed rare, showing how directly the design of the return determines the behavior an agent learns.