Epsilon-greedy (also written ε-greedy) is an action-selection strategy in reinforcement learning that chooses a uniformly random action with probability ε and the action with the highest estimated value with probability 1 − ε. It is the simplest widely used answer to the exploration-exploitation tradeoff: the agent mostly exploits what it already knows, yet keeps sampling alternatives so that its value estimates do not freeze around an early mistake.
The strategy appears in multi-armed bandits, in tabular Q-learning and SARSA, and in the Deep Q-network family, where a decaying ε schedule is part of the standard training recipe. Its appeal lies in one interpretable parameter and almost no computational overhead.
How epsilon-greedy selects an action
At every decision step the agent draws a random number between 0 and 1. If the number is below ε, the agent explores by picking any action uniformly at random; otherwise it exploits by taking the action with the largest current estimate, usually the Q-value Q(s, a) for the present state. Ties among greedy actions are normally broken at random to avoid a systematic bias toward the first-listed action.
Because the random branch may also select the greedy action, the greedy action’s total probability is 1 − ε + ε/|A|, where |A| is the number of available actions, and every other action receives ε/|A|. This guarantees that each action keeps a nonzero chance of being tried, which is the property convergence proofs for tabular methods rely on.
The table below shows how the resulting policy distributes probability for a small discrete action space of four actions.
| Epsilon value | Greedy action probability | Each other action | Behavior |
|---|---|---|---|
| ε = 0 | 100% | 0% | Pure exploitation |
| ε = 0.05 | 96.25% | 1.25% | Light exploration |
| ε = 0.1 | 92.5% | 2.5% | Common default |
| ε = 0.5 | 62.5% | 12.5% | Heavy exploration |
| ε = 1 | 25% | 25% | Uniform random policy |
Decay schedules and variants of epsilon-greedy
A fixed ε keeps exploring forever, which wastes reward once estimates are accurate, so most implementations use an annealing schedule that starts high and shrinks over time. The variant called GLIE, short for greedy in the limit with infinite exploration, requires that every action be tried infinitely often while the policy becomes greedy in the limit; a schedule such as ε = 1/t satisfies both conditions and supports convergence guarantees for tabular SARSA.
| Schedule | Rule | Typical settings | When it fits |
|---|---|---|---|
| Constant | ε stays fixed | 0.01 to 0.1 | Nonstationary environments |
| Linear decay | ε drops evenly to a floor | 1.0 to 0.1 or 0.01 over 1M steps | Deep Q-network training |
| Exponential decay | ε multiplied by a factor each episode | Factor 0.99 to 0.9999 | Tabular tasks with many episodes |
| Inverse time | ε = 1/t or c/t | Constant c near 1 | Theoretical GLIE guarantees |
| Per-actor fixed values | Each parallel actor uses its own ε | Spread from about 0.4 to near 0 | Distributed replay systems |
Other refinements include epsilon-first, which explores purely for a fixed budget and then exploits, and adaptive ε, which raises exploration when value estimates change quickly or when temporal difference errors grow. For evaluation runs, practitioners commonly use a small residual ε of about 0.001 to 0.05 so that a deterministic policy does not get stuck in repeating loops.
Epsilon-greedy compared with other exploration strategies
The defining weakness of ε-greedy is that its exploration is undirected: when it explores, a promising second-best action and an obviously terrible one are equally likely to be picked. More sophisticated strategies use value estimates or uncertainty to decide what is worth trying.
| Strategy | How it explores | Strength | Weakness |
|---|---|---|---|
| Epsilon-greedy | Uniform random action with probability ε | Trivial to implement and tune | Ignores how bad an action looks |
| Softmax (Boltzmann) | Samples in proportion to exp(Q/τ) | Prefers near-best actions | Sensitive to value scale |
| Upper confidence bound | Adds an uncertainty bonus to estimates | Strong bandit regret bounds | Hard to extend to deep networks |
| Thompson sampling | Samples from a posterior over values | Efficient, directed exploration | Needs a posterior model |
| Intrinsic motivation | Rewards novelty or prediction error | Works with sparse rewards | Extra models and tuning |
| Entropy regularization | Keeps stochastic policies spread out | Natural for policy gradient | Not applicable to pure value methods |
In bandit theory, a fixed ε yields linear regret, because a constant fraction of pulls is wasted, whereas decaying schedules and upper confidence bound methods achieve logarithmic regret. In deep reinforcement learning, however, ε-greedy remains competitive on many benchmarks because function approximation and experience replay smooth out its inefficiency.
Configuring epsilon-greedy for a task
Good configuration depends on episode length, action count and reward density. The decision table summarizes common situations and settings that experienced practitioners start from before tuning.
| Situation | Recommended setting | Reason |
|---|---|---|
| Small tabular problem | Exponential decay to about 0.01 | Estimates converge quickly |
| Atari-style Deep Q-network | Linear decay from 1.0 to 0.1 or 0.01 | Long warm-up fills the replay buffer |
| Drifting rewards or concept drift | Constant ε near 0.05 to 0.1 | Continued tracking of changes |
| Large action space | Consider softmax or structured exploration | Uniform random choices are mostly useless |
| Sparse reward with long horizons | Add intrinsic motivation | Random dithering rarely reaches the goal |
Because ε-greedy is a behavior policy that differs from the greedy target policy, it pairs naturally with off-policy learning such as Q-learning. With on-policy SARSA, the learned values include the cost of exploratory actions, which yields more cautious behavior near hazards, the effect illustrated by the well-known cliff-walking example.
Limitations and typical mistakes
The most serious limitation is inefficiency in deep exploration problems, where reaching a reward requires a long, specific sequence of actions. The probability that random dithering produces such a sequence falls exponentially with its length, so ε-greedy can fail completely on tasks that directed methods solve.
- Decaying ε too fast locks the agent into a suboptimal policy before value estimates are reliable.
- Forgetting to lower ε at evaluation time makes reported performance look worse than the learned policy.
- Uniform exploration in safety-critical settings can take dangerous actions; safe reinforcement learning adds constraints or action masking.
- Ignoring invalid actions wastes exploration; masking them before the random draw is standard practice.
- Comparing algorithms with different schedules confounds results, so the schedule should be reported alongside the method.
The simple explanation
Think of choosing where to eat lunch. Most days a person goes to the favorite restaurant, because it is reliably good. Once in a while, perhaps one day in ten, the person tries a random place instead. Sometimes the new place is worse, but occasionally it turns out to be a better new favorite.
Epsilon-greedy is that habit written as a rule. The number ε is how often to try something random. The key takeaway is that a learner who never experiments can stay stuck with a merely decent choice, while a little regular randomness keeps discovering better options.
How it is used in practice
A team training a Deep Q-network to play a classic arcade game sets ε to 1.0 at the start, so the agent acts completely at random while its replay buffer fills with the first 50,000 transitions. Over the next one million steps, ε falls linearly to 0.1 and then stays there, which keeps the agent sampling unfamiliar moves throughout training.
When the team measures the score, it switches to a nearly greedy policy with ε = 0.05 and averages thirty evaluation episodes. Early in training the agent scores barely above random play; after several million steps the evaluation score climbs steadily. When an experiment decays ε to 0.1 within only 100,000 steps, the agent settles on a weaker strategy and its final score is noticeably lower, which confirms that the decay schedule matters as much as the network design.