Hierarchical reinforcement learning

Also known as: HRL

On this page 7

Hierarchical reinforcement learning (HRL) is a family of methods in reinforcement learning that organizes an agent’s behavior into several levels of decision making, where a high-level policy chooses subgoals or skills and low-level policies execute them over many time steps. By replacing long chains of primitive actions with temporally extended actions, HRL attacks the two hardest problems of long-horizon tasks: sparse reward and the credit assignment problem.

In artificial intelligence research, HRL connects classical ideas about task decomposition with modern deep reinforcement learning. It is used in robotics, navigation, strategy games and agent systems wherever a task naturally breaks into stages that can be learned, reused and recombined.

How hierarchical reinforcement learning decomposes a task

HRL decomposes a task by letting a manager policy act on a coarse time scale and one or more worker policies act on a fine time scale. The manager observes the state, selects a subgoal or skill, and hands control to a worker that runs for several steps until a termination condition fires. Only then does the manager decide again.

This structure turns the original Markov decision process into a semi-Markov decision process (SMDP), because high-level decisions now last a variable number of steps. Rewards collected during a skill are accumulated and discounted as one transition, so the manager effectively sees a much shorter horizon. A task of 5,000 primitive steps may become a sequence of 20 to 50 high-level choices.

The benefit comes from three effects. Exploration becomes structured, since random choices among skills cover far more of the state space than random motor commands. Credit assignment shortens, because a reward needs to be propagated across fewer decisions. Skills transfer, because a learned behavior such as opening a door can serve many different tasks.

Main frameworks and architectures in HRL

Several formal frameworks define how levels communicate, and they differ mainly in what the high level outputs. The table below summarizes the frameworks that most published HRL methods build on.

FrameworkHigh-level outputKey ideaTypical limitation
Options frameworkAn option to executeOption = initiation set, policy, terminationOptions often hand-designed or collapse
MAXQ decompositionA subtask in a task graphValue function split into subtask valuesRequires a designer-specified hierarchy
Feudal networksA goal direction in latent spaceManager sets goals, worker is rewarded for reaching themLatent goals can be hard to interpret
Goal-conditioned HRLA target state or subgoalWorker learns a universal goal-reaching policyNonstationarity between levels
Skill discovery methodsA discrete or continuous skill codeSkills learned without task rewardDiscovered skills may be irrelevant to the task
Core HRL frameworks and what the high level selects

Well-known deep implementations include option-critic, which learns option policies and termination functions end to end with policy gradients, and HIRO, which uses off-policy learning for both levels and relabels high-level goals to correct for the changing worker. HAC (hierarchical actor-critic) stacks several goal-conditioned levels and uses hindsight relabeling to cope with sparse reward.

How HRL compares with flat reinforcement learning

Flat reinforcement learning maps each state directly to a primitive action, while HRL inserts intermediate abstractions between perception and control. The comparison below shows where each approach tends to win.

AspectFlat RLHierarchical RL
Decision horizonEvery primitive stepFew high-level decisions per episode
ExplorationLocal, action-level noiseTemporally coherent, skill-level
Sample efficiency on long tasksOften poor under sparse rewardOften much better when skills fit the task
TransferPolicy tied to one taskSkills reusable across tasks
Engineering complexityOne policy, one objectiveSeveral policies, interlocking objectives
OptimalityCan reach the optimal flat policyLimited by the chosen hierarchy
Flat versus hierarchical reinforcement learning

HRL also differs from related ideas in the same Topic. Reward shaping adds dense signals but keeps a single policy, while HRL changes the structure of the policy itself. Intrinsic motivation is frequently used inside HRL, for example to reward a worker for reaching a subgoal or to drive skill discovery. Model-based reinforcement learning plans with a learned dynamics model, and the two can be combined when a high level plans over skills instead of single actions.

Designing a hierarchy in practice

A practical HRL design answers three questions: where subgoals come from, how long each skill runs, and how the worker is rewarded. Subgoals can be hand-specified, chosen from states visited in the replay buffer, or produced as vectors in a learned representation space. Fixed skill durations of about 10 to 50 primitive steps are a common starting point in continuous control, while learned termination functions offer flexibility at the cost of stability.

SituationRecommended choiceReason
Task stages are known in advanceHand-designed options or MAXQInjects domain knowledge cheaply
Goals are positions or statesGoal-conditioned workerSubgoals are easy to measure and reward
No task reward during pretrainingUnsupervised skill discoveryBuilds a skill library before the real task
Data is expensive to collectOff-policy HRL with relabelingReuses old transitions despite changing levels
Short task with dense rewardFlat RLHierarchy adds overhead without benefit
Design choices and when to prefer each

Training usually starts with the worker, or trains both levels jointly with a lower learning rate for the manager. Many teams pretrain skills in simpler environments, freeze or fine-tune them, and then train the high level. Evaluating the hierarchy means checking not only total return but also skill diversity, skill duration and subgoal reachability.

Limitations and common failure modes

The central difficulty of HRL is nonstationarity between levels: while the worker is still learning, the same high-level command produces different outcomes, so the manager is learning from a moving target. Off-policy methods suffer most, which is why goal relabeling and careful update schedules are standard.

SymptomLikely causeCommon fix
Option collapseOne skill dominates, others unusedEntropy or diversity regularization
Degenerate terminationSkills end every step or never endDeliberation cost or fixed horizons
Unreachable subgoalsManager proposes impossible statesConstrain goals to visited states
Manager instabilityWorker behavior keeps changingHindsight relabeling, slower manager updates
Suboptimal final policyHierarchy restricts behaviorAllow primitive actions at the top level
Typical HRL failure modes with causes and remedies

There is also an honest research caveat: on many standard benchmarks, strong flat agents with good exploration match hierarchical methods, and gains from HRL are clearest on genuinely long-horizon, compositional tasks. Automatic discovery of useful hierarchies remains an open problem, so many successful systems still rely on partly designed structure.

The short version

Hierarchical reinforcement learning teaches a machine to work the way a person handles a big job: break it into steps, then do each step. Instead of deciding every tiny movement from scratch, the system has a planner that picks the next goal and a set of skills that know how to reach it.

Think of cooking dinner. Nobody plans each wrist movement; you decide to chop vegetables, then boil water, then set the table. Each step is a reusable skill, and the planner only chooses the order. This makes learning faster when rewards come rarely, as long as the steps fit the job.

Where it makes a difference

A warehouse robotics team trains a mobile manipulator to fetch items from shelves and place them in shipping bins. A flat agent receiving a reward only when an item lands in the correct bin almost never succeeds by chance, because a full episode lasts several thousand control steps.

The team instead pretrains three skills in simulation: navigate to a shelf location, grasp an object and place an object in a bin. A high-level policy chooses which skill to run and supplies its target, with each skill limited to about 200 control steps. Training the manager then requires decisions over only a handful of steps per episode.

The result is a robot that learns the full fetch task with far fewer simulated episodes than the flat baseline, and the navigation and grasping skills are reused later when the warehouse adds a restocking task.