Inverse reinforcement learning (IRL) is the problem of inferring the reward function that an agent is optimizing from observations of its behavior. Standard reinforcement learning starts from a known reward and searches for a good policy; IRL runs the problem backward, starting from expert demonstrations and asking which objective would make that behavior optimal or near-optimal.
IRL matters because, in many real tasks, writing a reward by hand is harder than demonstrating the task. A recovered reward is also more transferable than a copied policy: it captures why the expert acts, so an agent can re-plan when the environment, dynamics or starting state change. The field is closely linked to imitation learning, apprenticeship learning and the reward models used to align modern artificial intelligence systems.
How IRL recovers a reward from demonstrations
IRL works by searching for a reward under which the expert’s trajectories score higher than alternative behavior. The setting is a Markov decision process without its reward: states, actions, transition dynamics and a discount factor are known or can be sampled, and the learner receives a set of state-action trajectories produced by the expert.
Most algorithms represent the reward as a weighted combination of features, or as a neural network, and alternate two steps. First, they solve or approximate the forward reinforcement learning problem under the current reward. Second, they adjust the reward parameters so that the resulting policy behaves more like the expert, often by matching expected feature counts.
The table below lists the inputs and outputs of a typical IRL pipeline.
| Element | Description | Source |
|---|---|---|
| Expert demonstrations | Trajectories of states and actions | Human operators, logs or a trained agent |
| Environment model | Transition dynamics or a simulator | Known model or sampled interaction |
| Reward hypothesis class | Linear features or a neural network | Chosen by the designer |
| Inner RL solver | Finds the policy for a candidate reward | Value iteration, soft Q-learning, policy gradient |
| Recovered reward | Objective that explains the demonstrations | Output of the IRL algorithm |
Why the IRL problem is ill-posed
The central difficulty of IRL is reward ambiguity: many different rewards explain the same behavior. A constant reward of zero makes every policy optimal, and any potential-based shaping of a reward leaves the optimal policy unchanged. The data therefore never identifies one unique objective.
Each family of algorithms resolves this ambiguity with a different assumption. Max-margin methods require the expert to beat alternatives by a margin, maximum entropy IRL assumes the expert is noisily rational and picks the least committed distribution consistent with the data, and Bayesian IRL keeps a posterior over rewards instead of a single answer. The chosen assumption shapes which reward is returned, so it should match how the demonstrations were actually produced.
Main families of IRL algorithms
IRL methods differ in how they model expert optimality and how they scale. The next table summarizes the most influential approaches.
| Approach | Core idea | Strength | Limitation |
|---|---|---|---|
| Linear programming IRL | Finds rewards that make the expert policy optimal | Simple formulation | Assumes a fully optimal expert |
| Apprenticeship learning | Matches expected feature counts with a margin | Performance guarantees | Needs hand-designed features |
| Maximum entropy IRL | Treats trajectories as exponentially likely in reward | Handles suboptimal experts | Requires repeated inner RL |
| Bayesian IRL | Infers a posterior over reward functions | Quantifies uncertainty | Computationally heavy |
| Adversarial IRL | Trains a discriminator that doubles as a reward | Scales to continuous control | Training can be unstable |
Deep variants such as guided cost learning and adversarial inverse reinforcement learning replace linear features with neural networks and use sampling instead of exact planning. This makes IRL feasible in high-dimensional robotics, at the price of less interpretable rewards.
IRL compared with related approaches
IRL is one of several ways to learn from demonstrations, and it differs mainly in what it outputs. Behavioral cloning copies actions directly, while IRL recovers an objective that a separate reinforcement learning step then optimizes.
| Method | What it learns | Behavior under new conditions |
|---|---|---|
| Behavioral cloning | A direct mapping from state to action | Errors compound outside the demonstrated states |
| Inverse reinforcement learning | A reward function | Re-plans toward the inferred goal |
| Generative adversarial imitation learning | A policy matching expert occupancy | Robust, but yields no reusable reward |
| Reward shaping | Hand-written extra reward terms | Depends on the designer’s insight |
| Preference-based reward modeling | A reward from pairwise comparisons | Used for aligning language models |
Reward models trained from human preferences are a close relative of IRL: both treat human behavior or judgment as evidence about an unknown objective. The difference is the signal, comparisons rather than full demonstrations.
Practical limits and common mistakes
IRL is powerful but expensive and fragile in practice, because each outer iteration usually requires solving a reinforcement learning problem. Practitioners should plan for the following issues.
| Problem | Cause | Remedy |
|---|---|---|
| Reward overfits demonstrations | Too few or too narrow trajectories | Collect diverse demonstrations and regularize |
| Inner solver dominates cost | Full RL in every iteration | Use sampling-based or adversarial methods |
| Systematic expert bias learned | Humans are not perfectly rational | Model noise or known biases explicitly |
| Reward does not transfer | Reward entangled with the dynamics | Use disentangled methods and test in new settings |
| Reward hacking after deployment | Optimizer exploits gaps in the reward | Validate behavior, not only reward fit |
A frequent misconception is that a low imitation error proves the true intent was recovered. It does not: a reward can reproduce training behavior and still prefer harmful actions in unvisited states, so evaluation should include policies retrained under the learned reward in changed environments.
The idea in brief
Most learning agents are told what counts as success and then figure out how to achieve it. Inverse reinforcement learning works the other way around: it watches someone skilled and works out what they are trying to achieve.
Think of watching an experienced taxi driver for a week. Without asking, you notice that the driver avoids tolls, prefers wide roads and slows down near schools. From those habits you can describe their priorities. IRL does the same with data, and the inferred goal can then guide a machine in situations the driver never faced.
A practical scenario
A warehouse robotics team wants a mobile robot to navigate busy aisles the way experienced human forklift operators do. Writing the reward by hand proved difficult: penalties for distance, speed and closeness to people kept producing robots that were either reckless or frozen. The team instead recorded about 300 hours of operator trajectories with lidar and position data.
Using maximum entropy IRL with features such as distance to pedestrians, aisle width, turning rate and travel time, the team recovered weights showing that operators accepted a longer route of roughly 10 to 15 percent to keep at least two meters from workers. A policy trained under the learned reward in simulation was then tested on a new floor layout. Because it pursued the inferred priorities rather than copying routes, it adapted to the unfamiliar layout and produced fewer close passes than the earlier hand-tuned controller.