Model-free reinforcement learning

Also known as: model-free RL

On this page 7

Model-free reinforcement learning is the family of reinforcement learning methods in which an agent learns a policy, a value function or both directly from sampled experience, without building an explicit model of the environment’s transition probabilities and reward function. The agent improves its behavior by trial and error, using observed rewards and next states as the only learning signal.

Often shortened to model-free RL, this approach underlies Q-learning, deep Q-networks, policy gradient methods and proximal policy optimization, which makes it the dominant paradigm in practical artificial intelligence systems that learn to control, play or align models with feedback.

How model-free agents learn without a dynamics model

A model-free agent treats its task as a Markov decision process and records each step as a transition: the observed state, the chosen action, the resulting reward and the following state. Instead of estimating how the world works, it updates a value estimate or policy parameters so that actions leading to higher cumulative discounted reward become more likely.

The two main update principles are temporal difference (TD) learning, which bootstraps from the current estimate of the next state’s value, and Monte Carlo estimation, which waits until the end of an episode and uses the actual return. TD methods have lower variance but introduce bias, while Monte Carlo returns are unbiased but noisy.

The table below summarizes the quantities a model-free agent typically estimates.

QuantityWhat it estimatesUsed by
State value V(s)Expected return from a state under a policyActor-critic, TD(0)
Action value Q(s, a)Expected return after taking an actionQ-learning, SARSA, DQN
Advantage A(s, a)How much better an action is than averageA3C, PPO, GAE
Policy parametersProbability of each action in each stateREINFORCE, PPO, SAC
Quantities learned in model-free reinforcement learning

Value-based and policy-based families

Model-free methods divide into value-based, policy-based and actor-critic algorithms, depending on whether they learn values, a policy or both. Selecting a family is driven chiefly by the type of actions, discrete versus continuous, and by how much sample efficiency the project can afford to sacrifice.

FamilyRepresentative algorithmsAction spaceKey strength
Value-basedQ-learning, DQN, Double DQNDiscreteSimple and sample-reusing
Policy gradientREINFORCEDiscrete or continuousDirectly optimizes the objective
On-policy actor-criticA3C, TRPO, PPODiscrete or continuousStable, easy to scale
Off-policy actor-criticDDPG, TD3, SACContinuousHigher sample efficiency
Main families of model-free algorithms

Value-based methods select actions by taking the maximum over Q-values, which is cheap for a few discrete actions but impractical for continuous control. Policy-based methods output action distributions directly and handle continuous or stochastic actions naturally, at the cost of higher gradient variance.

Model-free compared with model-based reinforcement learning

The defining contrast is with model-based reinforcement learning, where the agent learns or is given a model of dynamics and uses it for planning, for example with Monte Carlo tree search or simulated rollouts. Model-free methods trade sample efficiency for simplicity and robustness.

CriterionModel-freeModel-based
Sample efficiencyLow; often millions of stepsHigh; can be 10 to 100 times fewer
Compute per stepLow at decision timeHigher because of planning
Model biasNone, learns from real returnsErrors compound in long rollouts
Asymptotic performanceOften strong with enough dataMay plateau below model-free
Implementation complexityModerateHigher, two coupled learners
Model-free versus model-based approaches

The boundary is not absolute. Experience replay acts as a nonparametric memory of transitions, and hybrid methods such as Dyna use a learned model to generate extra training data for a model-free learner.

As a rule of thumb, a model-free method is the sensible default when a fast, accurate simulator is available and interaction is cheap, as in games, robotics simulation and language model fine-tuning. When every real interaction is expensive, slow or risky, a model-based or offline approach usually deserves evaluation first, because the agent cannot afford millions of exploratory mistakes.

On-policy and off-policy learning in practice

Model-free algorithms also differ in whether they learn from data produced by the current policy. On-policy learning, as in SARSA and PPO, discards data after each update, while off-policy learning, as in Q-learning and SAC, reuses old transitions from a replay buffer and can learn from logged or demonstration data.

Typical hyperparameters strongly influence results, and their sensible ranges are well established in practice.

HyperparameterTypical rangeEffect when too high
Discount factor0.95 to 0.999Slow, high-variance credit assignment
Learning rate0.00001 to 0.001Divergence or policy collapse
Replay buffer size100,000 to 1,000,000 transitionsStale data from old policies
Epsilon for exploration1.0 decayed to 0.01-0.1Wasted steps on random actions
PPO clip range0.1 to 0.3Unstable, overly large policy updates
Common hyperparameters and typical ranges

Limitations and failure modes

Model-free reinforcement learning is powerful but demanding. Its weaknesses stem from learning only through sampled interaction, which makes training slow, noisy and sensitive to design choices.

  • Poor sample efficiency makes direct training on physical robots or costly systems impractical without a simulator.
  • The deadly triad, which combines bootstrapping, off-policy data and function approximation, can make value estimates diverge.
  • Overestimation bias in Q-learning inflates action values, which Double DQN and TD3 reduce with separate estimators.
  • Sparse rewards leave the agent with almost no signal, so reward shaping or intrinsic motivation becomes necessary.
  • Reward hacking occurs when the agent exploits flaws in the reward function instead of solving the intended task.
  • High seed sensitivity means results should be reported over several random seeds, not one lucky run.

Put simply

Model-free reinforcement learning is learning by doing, without a rulebook. The agent does not try to understand how its world works. It simply tries actions, notices which ones lead to rewards and gradually repeats the good ones more often.

Think of someone learning a new video game without reading the manual. They do not work out the game’s physics; they remember what paid off and adjust. This is simple and flexible, but it takes a great deal of practice before the behavior becomes reliable.

In practice

A data center operations team uses model-free reinforcement learning to tune cooling setpoints. A soft actor-critic agent observes about 30 sensor readings, including inlet temperatures, server load and outside air temperature, and adjusts fan speeds and chilled-water valves every five minutes. Its reward combines negative energy use with a heavy penalty for any rack exceeding its thermal limit.

The team first trains the agent in a calibrated simulator for several million steps, then fine-tunes it on logged plant data using off-policy updates, with a rule-based safety layer that overrides unsafe actions. After deployment, the controller reduces cooling energy by roughly 10 to 15 percent compared with the previous fixed schedule while keeping every rack within limits, and engineers retrain it each season as operating conditions drift.