Epsilon-greedy

Also known as: ε-greedy

On this page 7

Epsilon-greedy (also written ε-greedy) is an action-selection strategy in reinforcement learning that chooses a uniformly random action with probability ε and the action with the highest estimated value with probability 1 − ε. It is the simplest widely used answer to the exploration-exploitation tradeoff: the agent mostly exploits what it already knows, yet keeps sampling alternatives so that its value estimates do not freeze around an early mistake.

The strategy appears in multi-armed bandits, in tabular Q-learning and SARSA, and in the Deep Q-network family, where a decaying ε schedule is part of the standard training recipe. Its appeal lies in one interpretable parameter and almost no computational overhead.

How epsilon-greedy selects an action

At every decision step the agent draws a random number between 0 and 1. If the number is below ε, the agent explores by picking any action uniformly at random; otherwise it exploits by taking the action with the largest current estimate, usually the Q-value Q(s, a) for the present state. Ties among greedy actions are normally broken at random to avoid a systematic bias toward the first-listed action.

Because the random branch may also select the greedy action, the greedy action’s total probability is 1 − ε + ε/|A|, where |A| is the number of available actions, and every other action receives ε/|A|. This guarantees that each action keeps a nonzero chance of being tried, which is the property convergence proofs for tabular methods rely on.

The table below shows how the resulting policy distributes probability for a small discrete action space of four actions.

Epsilon valueGreedy action probabilityEach other actionBehavior
ε = 0100%0%Pure exploitation
ε = 0.0596.25%1.25%Light exploration
ε = 0.192.5%2.5%Common default
ε = 0.562.5%12.5%Heavy exploration
ε = 125%25%Uniform random policy
Action probabilities with four available actions

Decay schedules and variants of epsilon-greedy

A fixed ε keeps exploring forever, which wastes reward once estimates are accurate, so most implementations use an annealing schedule that starts high and shrinks over time. The variant called GLIE, short for greedy in the limit with infinite exploration, requires that every action be tried infinitely often while the policy becomes greedy in the limit; a schedule such as ε = 1/t satisfies both conditions and supports convergence guarantees for tabular SARSA.

ScheduleRuleTypical settingsWhen it fits
Constantε stays fixed0.01 to 0.1Nonstationary environments
Linear decayε drops evenly to a floor1.0 to 0.1 or 0.01 over 1M stepsDeep Q-network training
Exponential decayε multiplied by a factor each episodeFactor 0.99 to 0.9999Tabular tasks with many episodes
Inverse timeε = 1/t or c/tConstant c near 1Theoretical GLIE guarantees
Per-actor fixed valuesEach parallel actor uses its own εSpread from about 0.4 to near 0Distributed replay systems
Typical epsilon schedules

Other refinements include epsilon-first, which explores purely for a fixed budget and then exploits, and adaptive ε, which raises exploration when value estimates change quickly or when temporal difference errors grow. For evaluation runs, practitioners commonly use a small residual ε of about 0.001 to 0.05 so that a deterministic policy does not get stuck in repeating loops.

Epsilon-greedy compared with other exploration strategies

The defining weakness of ε-greedy is that its exploration is undirected: when it explores, a promising second-best action and an obviously terrible one are equally likely to be picked. More sophisticated strategies use value estimates or uncertainty to decide what is worth trying.

StrategyHow it exploresStrengthWeakness
Epsilon-greedyUniform random action with probability εTrivial to implement and tuneIgnores how bad an action looks
Softmax (Boltzmann)Samples in proportion to exp(Q/τ)Prefers near-best actionsSensitive to value scale
Upper confidence boundAdds an uncertainty bonus to estimatesStrong bandit regret boundsHard to extend to deep networks
Thompson samplingSamples from a posterior over valuesEfficient, directed explorationNeeds a posterior model
Intrinsic motivationRewards novelty or prediction errorWorks with sparse rewardsExtra models and tuning
Entropy regularizationKeeps stochastic policies spread outNatural for policy gradientNot applicable to pure value methods
Exploration strategies in reinforcement learning

In bandit theory, a fixed ε yields linear regret, because a constant fraction of pulls is wasted, whereas decaying schedules and upper confidence bound methods achieve logarithmic regret. In deep reinforcement learning, however, ε-greedy remains competitive on many benchmarks because function approximation and experience replay smooth out its inefficiency.

Configuring epsilon-greedy for a task

Good configuration depends on episode length, action count and reward density. The decision table summarizes common situations and settings that experienced practitioners start from before tuning.

SituationRecommended settingReason
Small tabular problemExponential decay to about 0.01Estimates converge quickly
Atari-style Deep Q-networkLinear decay from 1.0 to 0.1 or 0.01Long warm-up fills the replay buffer
Drifting rewards or concept driftConstant ε near 0.05 to 0.1Continued tracking of changes
Large action spaceConsider softmax or structured explorationUniform random choices are mostly useless
Sparse reward with long horizonsAdd intrinsic motivationRandom dithering rarely reaches the goal
Starting points for epsilon configuration

Because ε-greedy is a behavior policy that differs from the greedy target policy, it pairs naturally with off-policy learning such as Q-learning. With on-policy SARSA, the learned values include the cost of exploratory actions, which yields more cautious behavior near hazards, the effect illustrated by the well-known cliff-walking example.

Limitations and typical mistakes

The most serious limitation is inefficiency in deep exploration problems, where reaching a reward requires a long, specific sequence of actions. The probability that random dithering produces such a sequence falls exponentially with its length, so ε-greedy can fail completely on tasks that directed methods solve.

  • Decaying ε too fast locks the agent into a suboptimal policy before value estimates are reliable.
  • Forgetting to lower ε at evaluation time makes reported performance look worse than the learned policy.
  • Uniform exploration in safety-critical settings can take dangerous actions; safe reinforcement learning adds constraints or action masking.
  • Ignoring invalid actions wastes exploration; masking them before the random draw is standard practice.
  • Comparing algorithms with different schedules confounds results, so the schedule should be reported alongside the method.

The simple explanation

Think of choosing where to eat lunch. Most days a person goes to the favorite restaurant, because it is reliably good. Once in a while, perhaps one day in ten, the person tries a random place instead. Sometimes the new place is worse, but occasionally it turns out to be a better new favorite.

Epsilon-greedy is that habit written as a rule. The number ε is how often to try something random. The key takeaway is that a learner who never experiments can stay stuck with a merely decent choice, while a little regular randomness keeps discovering better options.

How it is used in practice

A team training a Deep Q-network to play a classic arcade game sets ε to 1.0 at the start, so the agent acts completely at random while its replay buffer fills with the first 50,000 transitions. Over the next one million steps, ε falls linearly to 0.1 and then stays there, which keeps the agent sampling unfamiliar moves throughout training.

When the team measures the score, it switches to a nearly greedy policy with ε = 0.05 and averages thirty evaluation episodes. Early in training the agent scores barely above random play; after several million steps the evaluation score climbs steadily. When an experiment decays ε to 0.1 within only 100,000 steps, the agent settles on a weaker strategy and its final score is noticeably lower, which confirms that the decay schedule matters as much as the network design.