Inverse reinforcement learning

Also known as: IRL

On this page 7

Inverse reinforcement learning (IRL) is the problem of inferring the reward function that an agent is optimizing from observations of its behavior. Standard reinforcement learning starts from a known reward and searches for a good policy; IRL runs the problem backward, starting from expert demonstrations and asking which objective would make that behavior optimal or near-optimal.

IRL matters because, in many real tasks, writing a reward by hand is harder than demonstrating the task. A recovered reward is also more transferable than a copied policy: it captures why the expert acts, so an agent can re-plan when the environment, dynamics or starting state change. The field is closely linked to imitation learning, apprenticeship learning and the reward models used to align modern artificial intelligence systems.

How IRL recovers a reward from demonstrations

IRL works by searching for a reward under which the expert’s trajectories score higher than alternative behavior. The setting is a Markov decision process without its reward: states, actions, transition dynamics and a discount factor are known or can be sampled, and the learner receives a set of state-action trajectories produced by the expert.

Most algorithms represent the reward as a weighted combination of features, or as a neural network, and alternate two steps. First, they solve or approximate the forward reinforcement learning problem under the current reward. Second, they adjust the reward parameters so that the resulting policy behaves more like the expert, often by matching expected feature counts.

The table below lists the inputs and outputs of a typical IRL pipeline.

ElementDescriptionSource
Expert demonstrationsTrajectories of states and actionsHuman operators, logs or a trained agent
Environment modelTransition dynamics or a simulatorKnown model or sampled interaction
Reward hypothesis classLinear features or a neural networkChosen by the designer
Inner RL solverFinds the policy for a candidate rewardValue iteration, soft Q-learning, policy gradient
Recovered rewardObjective that explains the demonstrationsOutput of the IRL algorithm
Inputs and outputs of an IRL pipeline

Why the IRL problem is ill-posed

The central difficulty of IRL is reward ambiguity: many different rewards explain the same behavior. A constant reward of zero makes every policy optimal, and any potential-based shaping of a reward leaves the optimal policy unchanged. The data therefore never identifies one unique objective.

Each family of algorithms resolves this ambiguity with a different assumption. Max-margin methods require the expert to beat alternatives by a margin, maximum entropy IRL assumes the expert is noisily rational and picks the least committed distribution consistent with the data, and Bayesian IRL keeps a posterior over rewards instead of a single answer. The chosen assumption shapes which reward is returned, so it should match how the demonstrations were actually produced.

Main families of IRL algorithms

IRL methods differ in how they model expert optimality and how they scale. The next table summarizes the most influential approaches.

ApproachCore ideaStrengthLimitation
Linear programming IRLFinds rewards that make the expert policy optimalSimple formulationAssumes a fully optimal expert
Apprenticeship learningMatches expected feature counts with a marginPerformance guaranteesNeeds hand-designed features
Maximum entropy IRLTreats trajectories as exponentially likely in rewardHandles suboptimal expertsRequires repeated inner RL
Bayesian IRLInfers a posterior over reward functionsQuantifies uncertaintyComputationally heavy
Adversarial IRLTrains a discriminator that doubles as a rewardScales to continuous controlTraining can be unstable
Major IRL approaches

Deep variants such as guided cost learning and adversarial inverse reinforcement learning replace linear features with neural networks and use sampling instead of exact planning. This makes IRL feasible in high-dimensional robotics, at the price of less interpretable rewards.

IRL is one of several ways to learn from demonstrations, and it differs mainly in what it outputs. Behavioral cloning copies actions directly, while IRL recovers an objective that a separate reinforcement learning step then optimizes.

MethodWhat it learnsBehavior under new conditions
Behavioral cloningA direct mapping from state to actionErrors compound outside the demonstrated states
Inverse reinforcement learningA reward functionRe-plans toward the inferred goal
Generative adversarial imitation learningA policy matching expert occupancyRobust, but yields no reusable reward
Reward shapingHand-written extra reward termsDepends on the designer’s insight
Preference-based reward modelingA reward from pairwise comparisonsUsed for aligning language models
IRL and neighboring methods

Reward models trained from human preferences are a close relative of IRL: both treat human behavior or judgment as evidence about an unknown objective. The difference is the signal, comparisons rather than full demonstrations.

Practical limits and common mistakes

IRL is powerful but expensive and fragile in practice, because each outer iteration usually requires solving a reinforcement learning problem. Practitioners should plan for the following issues.

ProblemCauseRemedy
Reward overfits demonstrationsToo few or too narrow trajectoriesCollect diverse demonstrations and regularize
Inner solver dominates costFull RL in every iterationUse sampling-based or adversarial methods
Systematic expert bias learnedHumans are not perfectly rationalModel noise or known biases explicitly
Reward does not transferReward entangled with the dynamicsUse disentangled methods and test in new settings
Reward hacking after deploymentOptimizer exploits gaps in the rewardValidate behavior, not only reward fit
Typical problems and remedies

A frequent misconception is that a low imitation error proves the true intent was recovered. It does not: a reward can reproduce training behavior and still prefer harmful actions in unvisited states, so evaluation should include policies retrained under the learned reward in changed environments.

The idea in brief

Most learning agents are told what counts as success and then figure out how to achieve it. Inverse reinforcement learning works the other way around: it watches someone skilled and works out what they are trying to achieve.

Think of watching an experienced taxi driver for a week. Without asking, you notice that the driver avoids tolls, prefers wide roads and slows down near schools. From those habits you can describe their priorities. IRL does the same with data, and the inferred goal can then guide a machine in situations the driver never faced.

A practical scenario

A warehouse robotics team wants a mobile robot to navigate busy aisles the way experienced human forklift operators do. Writing the reward by hand proved difficult: penalties for distance, speed and closeness to people kept producing robots that were either reckless or frozen. The team instead recorded about 300 hours of operator trajectories with lidar and position data.

Using maximum entropy IRL with features such as distance to pedestrians, aisle width, turning rate and travel time, the team recovered weights showing that operators accepted a longer route of roughly 10 to 15 percent to keep at least two meters from workers. A policy trained under the learned reward in simulation was then tested on a new floor layout. Because it pursued the inferred priorities rather than copying routes, it adapted to the unfamiliar layout and produced fewer close passes than the earlier hand-tuned controller.