Episode · reinforcement learning

On this page 7

An episode in reinforcement learning is one complete sequence of interaction between an agent and its environment, starting in an initial state and ending when the environment reaches a terminal state or a time limit cuts it off. Each episode produces a trajectory of states, actions and rewards, and the sum of those rewards, usually discounted, is the return that the agent tries to maximize.

The episode is the basic unit of experience in artificial intelligence systems that learn by trial and error. A game of chess, a single attempt by a robot arm to grasp an object, or one simulated drive from start to destination are all episodes. Whether a task is episodic or continuing shapes how returns are defined, which algorithms apply and how progress is measured.

Structure of an episode

An episode consists of a sequence of discrete time steps, and at each step the agent observes a state, chooses an action and receives a reward together with the next state. Formally, the trajectory is written as s0, a0, r1, s1, a1, r2 and so on until a final step T, where the episode resets to a new initial state drawn from a start distribution.

ComponentMeaningExample in a maze task
Initial stateState sampled at resetAgent placed at the entrance
Time stepOne observe, act, reward cycleOne move on the grid
TrajectoryOrdered record of the whole runFull path through the maze
Terminal stateState that ends the episodeExit reached or trap entered
Episode lengthNumber of steps TBetween 12 and 500 moves
ReturnDiscounted sum of rewardsPlus 1 minus a small step penalty
Components of a single episode

In the language of a Markov decision process, a terminal state is an absorbing state: once entered, all further rewards are zero and no transition leaves it. This convention lets the same Bellman equation describe both episodic and continuing tasks, because the value of any terminal state is defined as exactly zero.

Episodic and continuing tasks

Reinforcement learning tasks fall into two families: episodic tasks that break naturally into separate episodes and continuing tasks that run indefinitely without a natural end. The distinction decides whether the return is a finite sum and whether a discount factor below 1 is mathematically required.

PropertyEpisodic taskContinuing task
End conditionTerminal state or time limitNone
Return definitionFinite sum to step TInfinite discounted sum
Discount factorMay equal 1Must be below 1, or use average reward
Typical examplesBoard games, robot graspingProcess control, recommendation feeds
Performance metricMean return per episodeAverage reward per step
Episodic versus continuing tasks

Many practical problems are continuing by nature but are artificially divided into episodes for training convenience. A data center cooling controller, for example, never stops, yet simulation runs of one simulated day are treated as episodes. Designers must then decide whether the end of such a run is a true termination or merely a cut, which leads directly to the next issue.

Termination versus truncation

An episode can end for two very different reasons, and confusing them is one of the most common implementation bugs in deep reinforcement learning. Termination means the environment reached a genuine terminal state, so no future reward exists. Truncation means an external limit, such as a maximum of 1,000 steps, stopped the run while the task could have continued.

Ending typeCauseCorrect bootstrap target
TerminationGoal reached, failure, game overReward only, next value set to zero
TruncationStep limit or wall-clock limitReward plus discounted value of next state
Environment reset by errorSimulator crash or invalid stateDiscard or treat as truncation
How termination and truncation affect learning targets

Treating truncation as termination tells a value-based method such as Q-learning or a deep Q-network that the future is worthless at the time limit. The agent then learns distorted value estimates and may behave erratically near the step limit. Modern environment interfaces, including the Gymnasium API, therefore return separate terminated and truncated flags so that algorithms can bootstrap correctly.

How algorithms use episodes

Learning algorithms differ in how much of an episode they need before they can update. Monte Carlo methods wait for the full episode to compute an exact return, while temporal difference learning updates after every step using a bootstrapped estimate.

Algorithm familyUpdate timingNeeds complete episodes
Monte Carlo methodsAfter the episode endsYes
REINFORCEAfter the episode endsYes, in its basic form
Temporal difference learningEvery stepNo
SARSA and Q-learningEvery stepNo
Proximal policy optimizationAfter fixed-length rolloutsNo, partial episodes allowed
Episode requirements of common algorithm families

Episode length also drives the credit assignment problem. In a long episode with a single reward at the end, a sparse reward must be traced back through hundreds of actions, which inflates the variance of Monte Carlo returns. Techniques such as reward shaping, n-step returns and generalized advantage estimation shorten this effective horizon.

Practical design decisions

Defining episodes well is an environment design decision with a direct effect on what the agent learns. A poorly chosen time limit or reset rule can teach the agent shortcuts that do not transfer to the real task.

  • Choose a maximum episode length comfortably above the steps an expert policy needs, often two to five times longer.
  • Randomize initial states so the agent does not memorize one starting configuration.
  • Do not include the remaining time in the state unless the real task has a deadline, because the agent may exploit it.
  • Report results as mean return over many evaluation episodes, with spread across random seeds, rather than a single best run.

Early termination on failure, such as ending a walking episode when the robot falls, speeds up training by avoiding wasted steps. It can also bias behavior: if each step yields positive reward, the agent avoids termination, but if each step costs reward, the agent may learn to end episodes deliberately to stop the penalty.

Understanding it simply

An episode is one round of practice. Think of a person learning to play a video game level: each attempt starts at the beginning and ends when they win, lose, or run out of time. Each attempt is one episode, and after it the level starts again.

The learning program works the same way. It plays many rounds, adds up the points it earned in each one, and slowly figures out which choices lead to higher scores across a whole round, not just in the next moment.

Applied in practice

A robotics team trains a simulated arm to pick up boxes from a conveyor belt. Each episode begins with a box placed at a random position and orientation and ends either when the box is lifted 10 centimeters, which counts as termination with a reward of plus 10, or after 200 control steps, which counts as truncation.

In the first version, the code treated both endings as terminal. The arm learned to move slowly and rarely finished, because the value function believed nothing mattered after step 200. After the team separated terminated and truncated flags, the success rate in evaluation rose from about 55 percent to over 90 percent across 1,000 test episodes, and the trained policy transferred to the physical arm with far less retuning.