The discount factor, written as the Greek letter gamma (γ), is a number between 0 and 1 that determines how much a reinforcement learning agent values future rewards relative to immediate rewards. A reward received k steps in the future is multiplied by γ raised to the power k before it enters the agent’s estimate of return.
In reinforcement learning, the discount factor is part of the definition of a Markov decision process and therefore shapes every value function, Q-function and policy the agent learns. It is sometimes called the discount rate, although strictly speaking a rate r relates to the factor through γ = 1/(1 + r), the same convention used in finance.
How the discount factor shapes the return
The agent’s objective is to maximize the discounted cumulative reward, also called the return. At time step t the return equals the immediate reward plus γ times the next reward, plus γ squared times the reward after that, and so on to the end of the episode or to infinity.
Discounting serves two distinct purposes. Mathematically, a value of γ below 1 guarantees that an infinite sum of bounded rewards stays finite, which makes continuing tasks well defined. Behaviorally, it encodes a preference for earlier rewards, reflecting uncertainty about the future or a genuine cost of delay.
The discount factor appears directly in the Bellman equation, where the value of a state equals the expected immediate reward plus γ times the value of the next state. This recursive structure is what allows temporal difference learning, Q-learning and value iteration to update estimates from one step of experience at a time.
The weight assigned to future rewards decays geometrically. The following table shows how quickly that weight falls for several common values of gamma.
| Gamma | After 10 steps | After 100 steps | After 1,000 steps |
|---|---|---|---|
| 0.5 | 0.001 | Practically zero | Practically zero |
| 0.9 | 0.35 | 0.00003 | Practically zero |
| 0.99 | 0.90 | 0.37 | 0.00004 |
| 0.999 | 0.99 | 0.90 | 0.37 |
The effective horizon of an agent
A useful rule of thumb is that the discount factor defines an effective planning horizon of about 1/(1 − γ) steps. Rewards much further away than this horizon contribute little to the value estimate, so the agent behaves as if it were planning over a window of roughly that length.
| Gamma | Effective horizon | Agent behavior | Typical application |
|---|---|---|---|
| 0 | 1 step | Purely greedy | Contextual bandit problems |
| 0.9 | About 10 steps | Short-sighted | Simple grid worlds, fast prototyping |
| 0.99 | About 100 steps | Balanced | Atari games, most deep RL benchmarks |
| 0.995 to 0.999 | 200 to 1,000 steps | Far-sighted | Robotics, long-horizon control |
| 1.0 | Whole episode | Undiscounted | Short episodic tasks with guaranteed termination |
Matching gamma to the time scale of the task is essential. If an agent controls a robot at 50 decisions per second and a meaningful outcome arrives after 10 seconds, a gamma of 0.99 with a horizon of about 100 steps would make the agent blind to the outcome it is supposed to pursue.
Discount factor versus related concepts
The discount factor is often confused with other parameters that also lie between 0 and 1. Distinguishing them matters because each controls a different aspect of learning, and tuning one cannot compensate for a poorly chosen other.
| Parameter | Symbol | What it controls |
|---|---|---|
| Discount factor | γ | Relative value of future rewards in the objective |
| Learning rate | α | Step size of each value or policy update |
| Exploration rate | ε | Probability of a random action in epsilon-greedy |
| Trace decay | λ | Bias-variance balance in eligibility traces and generalized advantage estimation |
| Target update rate | τ | Speed of soft updates to target networks |
The pairing of γ and λ in generalized advantage estimation deserves special attention. Gamma changes the problem being solved, while lambda only changes how the advantage is estimated for that problem, so practitioners such as those using proximal policy optimization usually keep gamma near 0.99 and lambda near 0.95.
Choosing a discount factor in practice
Choosing gamma is a design decision about the objective, not only a tuning knob. A higher value aligns the learned policy more closely with long-term success but increases the variance of return estimates and slows convergence, because errors propagate across more steps.
- Start with 0.99 for most episodic tasks of a few hundred to a few thousand steps.
- Raise gamma when important rewards are sparse and arrive long after the actions that cause them.
- Lower gamma when training is unstable, the environment is highly stochastic or a short horizon is genuinely desired.
- Treat gamma as part of the task specification when comparing algorithms, because results at different gammas are not comparable.
- Consider reward shaping or hierarchical reinforcement learning before pushing gamma very close to 1.
Common pitfalls with discounting
Several recurring mistakes stem from treating the discount factor as a minor detail. The table pairs typical symptoms with their most likely cause and a corrective step.
| Symptom | Likely cause | Fix |
|---|---|---|
| Agent ignores a distant goal | Gamma too low for the task horizon | Increase gamma or shorten the time step |
| Value estimates diverge or oscillate | Gamma too close to 1 with bootstrapping | Lower gamma or stabilize targets |
| Agent delays reaching the goal | Undiscounted positive step rewards | Use gamma below 1 or a step penalty |
| Policy loops forever in continuing task | Gamma of 1 without termination | Use gamma below 1 or average reward |
A subtler issue is that many implementations train with discounting but evaluate with undiscounted returns. This mismatch is common and often acceptable, but it means the agent is optimizing a slightly different objective from the one reported in benchmarks, which practitioners should state explicitly.
The simple explanation
The discount factor answers a simple question: how much is a reward tomorrow worth compared with a reward today? If the answer is “almost as much,” the agent is patient and plans ahead. If the answer is “hardly anything,” the agent grabs whatever it can right now.
It works much like interest on money. A dollar promised in ten years feels worth less than a dollar in hand. A gamma close to 1 makes the agent patient; a gamma close to 0 makes it impatient. Picking the right value tells the agent how far ahead it should look.
How it is used in practice
A logistics company trains a reinforcement learning agent to control the temperature of a refrigerated warehouse. The agent decides every five minutes whether to run compressors, and electricity prices change hourly, so a cheap decision now can cause an expensive temperature recovery several hours later.
An initial version uses gamma of 0.9, giving a horizon of only about ten decisions, or under one hour. The agent saves energy in the short term but repeatedly lets the temperature drift toward its upper limit before evening price peaks. Engineers raise gamma to 0.995, which stretches the horizon to about 200 decisions, roughly 16 hours.
With the longer horizon the agent learns to precool the warehouse during cheap night hours and coast through expensive afternoon periods. Energy costs fall noticeably while stored goods remain within the required temperature range, at the price of a longer and more carefully monitored training run.