Discount factor

Also known as: discount rate

On this page 7

The discount factor, written as the Greek letter gamma (γ), is a number between 0 and 1 that determines how much a reinforcement learning agent values future rewards relative to immediate rewards. A reward received k steps in the future is multiplied by γ raised to the power k before it enters the agent’s estimate of return.

In reinforcement learning, the discount factor is part of the definition of a Markov decision process and therefore shapes every value function, Q-function and policy the agent learns. It is sometimes called the discount rate, although strictly speaking a rate r relates to the factor through γ = 1/(1 + r), the same convention used in finance.

How the discount factor shapes the return

The agent’s objective is to maximize the discounted cumulative reward, also called the return. At time step t the return equals the immediate reward plus γ times the next reward, plus γ squared times the reward after that, and so on to the end of the episode or to infinity.

Discounting serves two distinct purposes. Mathematically, a value of γ below 1 guarantees that an infinite sum of bounded rewards stays finite, which makes continuing tasks well defined. Behaviorally, it encodes a preference for earlier rewards, reflecting uncertainty about the future or a genuine cost of delay.

The discount factor appears directly in the Bellman equation, where the value of a state equals the expected immediate reward plus γ times the value of the next state. This recursive structure is what allows temporal difference learning, Q-learning and value iteration to update estimates from one step of experience at a time.

The weight assigned to future rewards decays geometrically. The following table shows how quickly that weight falls for several common values of gamma.

GammaAfter 10 stepsAfter 100 stepsAfter 1,000 steps
0.50.001Practically zeroPractically zero
0.90.350.00003Practically zero
0.990.900.370.00004
0.9990.990.900.37
Weight of a reward received after k steps for different gamma values

The effective horizon of an agent

A useful rule of thumb is that the discount factor defines an effective planning horizon of about 1/(1 − γ) steps. Rewards much further away than this horizon contribute little to the value estimate, so the agent behaves as if it were planning over a window of roughly that length.

GammaEffective horizonAgent behaviorTypical application
01 stepPurely greedyContextual bandit problems
0.9About 10 stepsShort-sightedSimple grid worlds, fast prototyping
0.99About 100 stepsBalancedAtari games, most deep RL benchmarks
0.995 to 0.999200 to 1,000 stepsFar-sightedRobotics, long-horizon control
1.0Whole episodeUndiscountedShort episodic tasks with guaranteed termination
Gamma values, effective horizons and typical uses

Matching gamma to the time scale of the task is essential. If an agent controls a robot at 50 decisions per second and a meaningful outcome arrives after 10 seconds, a gamma of 0.99 with a horizon of about 100 steps would make the agent blind to the outcome it is supposed to pursue.

The discount factor is often confused with other parameters that also lie between 0 and 1. Distinguishing them matters because each controls a different aspect of learning, and tuning one cannot compensate for a poorly chosen other.

ParameterSymbolWhat it controls
Discount factorγRelative value of future rewards in the objective
Learning rateαStep size of each value or policy update
Exploration rateεProbability of a random action in epsilon-greedy
Trace decayλBias-variance balance in eligibility traces and generalized advantage estimation
Target update rateτSpeed of soft updates to target networks
Parameters commonly confused with the discount factor

The pairing of γ and λ in generalized advantage estimation deserves special attention. Gamma changes the problem being solved, while lambda only changes how the advantage is estimated for that problem, so practitioners such as those using proximal policy optimization usually keep gamma near 0.99 and lambda near 0.95.

Choosing a discount factor in practice

Choosing gamma is a design decision about the objective, not only a tuning knob. A higher value aligns the learned policy more closely with long-term success but increases the variance of return estimates and slows convergence, because errors propagate across more steps.

  • Start with 0.99 for most episodic tasks of a few hundred to a few thousand steps.
  • Raise gamma when important rewards are sparse and arrive long after the actions that cause them.
  • Lower gamma when training is unstable, the environment is highly stochastic or a short horizon is genuinely desired.
  • Treat gamma as part of the task specification when comparing algorithms, because results at different gammas are not comparable.
  • Consider reward shaping or hierarchical reinforcement learning before pushing gamma very close to 1.

Common pitfalls with discounting

Several recurring mistakes stem from treating the discount factor as a minor detail. The table pairs typical symptoms with their most likely cause and a corrective step.

SymptomLikely causeFix
Agent ignores a distant goalGamma too low for the task horizonIncrease gamma or shorten the time step
Value estimates diverge or oscillateGamma too close to 1 with bootstrappingLower gamma or stabilize targets
Agent delays reaching the goalUndiscounted positive step rewardsUse gamma below 1 or a step penalty
Policy loops forever in continuing taskGamma of 1 without terminationUse gamma below 1 or average reward
Symptoms, causes and fixes related to gamma

A subtler issue is that many implementations train with discounting but evaluate with undiscounted returns. This mismatch is common and often acceptable, but it means the agent is optimizing a slightly different objective from the one reported in benchmarks, which practitioners should state explicitly.

The simple explanation

The discount factor answers a simple question: how much is a reward tomorrow worth compared with a reward today? If the answer is “almost as much,” the agent is patient and plans ahead. If the answer is “hardly anything,” the agent grabs whatever it can right now.

It works much like interest on money. A dollar promised in ten years feels worth less than a dollar in hand. A gamma close to 1 makes the agent patient; a gamma close to 0 makes it impatient. Picking the right value tells the agent how far ahead it should look.

How it is used in practice

A logistics company trains a reinforcement learning agent to control the temperature of a refrigerated warehouse. The agent decides every five minutes whether to run compressors, and electricity prices change hourly, so a cheap decision now can cause an expensive temperature recovery several hours later.

An initial version uses gamma of 0.9, giving a horizon of only about ten decisions, or under one hour. The agent saves energy in the short term but repeatedly lets the temperature drift toward its upper limit before evening price peaks. Engineers raise gamma to 0.995, which stretches the horizon to about 200 decisions, roughly 16 hours.

With the longer horizon the agent learns to precool the warehouse during cheap night hours and coast through expensive afternoon periods. Energy costs fall noticeably while stored goods remain within the required temperature range, at the price of a longer and more carefully monitored training run.