Q-function

Also known as: action-value function

On this page 7

The Q-function, also called the action-value function, assigns to every state and action the expected discounted return an agent obtains if it takes that action in that state and then follows a given policy. In reinforcement learning, it is the central quantity behind value-based control, because an agent that knows the optimal Q-function can act optimally by choosing the highest-valued action.

Written as Qπ(s, a) for a policy π and Q*(s, a) for the optimal policy, the Q-function connects the reward function, the discount factor and the policy into a single number per state-action pair. Algorithms such as Q-learning, SARSA and the Deep Q-network all exist to estimate it.

How the Q-function is defined

Formally, Qπ(s, a) = E[G_t | S_t = s, A_t = a], where the return G_t is the sum of future rewards discounted by γ at each step. The expectation runs over the randomness of the environment transitions and of the policy π after the first action, which is fixed to a.

The Q-function satisfies a recursive Bellman equation. For a fixed policy, Qπ(s, a) equals the immediate expected reward plus γ times the expected Q-value of the next state-action pair under π. For the optimal policy, the Bellman optimality equation replaces that expectation with a maximum over next actions, which is the update target used by Q-learning.

The following table lists the symbols that appear in Q-function definitions and their usual ranges.

SymbolMeaningTypical range or form
s, aState and actionDiscrete indices or continuous vectors
rImmediate rewardAny real number defined by the task
γ (gamma)Discount factorUsually 0.9 to 0.999
Qπ(s, a)Action value under policy πExpected discounted return
Q*(s, a)Optimal action valueMaximum achievable expected return
Notation used with the Q-function

Q-function compared with other value functions

The Q-function differs from the state-value function V(s) by conditioning on the first action, which makes it directly usable for decisions. With V alone, choosing an action requires a model of transitions and rewards; with Q, the agent simply picks argmax over a, which is why model-free control methods prefer it.

The three functions are tightly linked: V(s) is the policy-weighted average of Q(s, a) over actions, and the advantage is the difference between the two. Knowing these identities helps when reading actor-critic and policy gradient methods, which often estimate one function and derive another.

FunctionInputWhat it answersMain use
State-value V(s)StateHow good is this state under πCritic baselines, planning
Action-value Q(s, a)State and actionHow good is this action hereAction selection, Q-learning
Advantage A(s, a)State and actionHow much better than averageVariance reduction in policy gradients
Q-function, value function and advantage function

How algorithms estimate the Q-function

Q-function estimation methods differ mainly in which target they bootstrap toward and in how they represent Q. Small problems use a tabular Q-table with one entry per state-action pair; large or continuous problems use a function approximator, usually a neural network.

The update target defines whether a method learns about the policy it follows or about the greedy policy. Q-learning is off-policy because its target uses the maximum next Q-value, while SARSA is on-policy because its target uses the action the agent actually took next.

AlgorithmUpdate targetPolicy typeRepresentation
Q-learningr + γ max Q(s’, a’)Off-policyTable or approximator
SARSAr + γ Q(s’, a’) with taken a’On-policyTable or approximator
Deep Q-networkr + γ max Q_target(s’, a’)Off-policyNeural network, replay buffer
Double DQNOnline net selects, target net evaluatesOff-policyNeural network
Monte Carlo controlFull observed return G_tOn-policy or off-policyTable or approximator
Main algorithms that learn a Q-function

Actor-critic methods for continuous actions, such as DDPG, TD3 and soft actor-critic, also learn a Q-function as the critic. Because the maximum over a continuous action space is hard to compute, a separate actor network proposes the action that the critic evaluates.

Using the Q-function to choose actions

A learned Q-function becomes a policy through an action selection rule. The purely greedy rule takes the highest-valued action, but during learning the agent must also explore, so practical systems use rules that sometimes choose actions with lower estimated value.

RuleHow it uses QWhen it fits
GreedyAlways picks argmax Q(s, a)Deployment after training
Epsilon-greedyRandom action with probability εSimple discrete tasks, DQN training
Softmax (Boltzmann)Probabilities proportional to exp(Q/τ)When value gaps should guide exploration
Upper confidence boundQ plus an uncertainty bonusBandits and tree search
Action selection rules built on Q-values

In practice, ε is usually annealed from about 1.0 to 0.01 or 0.1 over training, so the agent explores widely at first and exploits its estimates later. The schedule matters as much as the network architecture in many benchmark results.

Limitations and common pitfalls

Q-function learning is powerful but fragile, especially with function approximation. The combination of bootstrapping, off-policy data and function approximation, often called the deadly triad, can make estimates diverge instead of converging.

  • Overestimation bias arises because the max operator picks noisy high estimates; Double Q-learning and Double DQN reduce it.
  • Tabular methods do not scale: the table grows with the product of state and action counts.
  • Continuous actions make the max over a expensive, which is why actor-critic designs are used there.
  • Sparse rewards leave most Q-values uninformative for long periods, slowing learning.
  • Offline data can contain actions never tried, and Q-values for those actions are often wildly optimistic.

Stabilizing tricks such as target networks, experience replay and reward clipping are standard in deep variants. Even with them, results can vary substantially between random seeds, so careful evaluation across several runs is necessary.

In everyday language

The Q-function is like a scorecard for choices. For every situation an agent can be in and every move it could make, the scorecard says how much total reward that move is likely to bring in the long run.

Imagine standing at a crossroads with a guidebook that lists, for each road, how pleasant the whole rest of the trip will be. You do not need to know the map in detail; you just pick the road with the best score. A reinforcement learning agent builds this guidebook by trial and error, improving the scores each time it sees what actually happens.

Seen in the real world

An operations team at a large data center trains an agent to adjust cooling setpoints every five minutes. The state includes server load, outdoor temperature and current rack temperatures, and the agent chooses among nine discrete setpoint changes. The reward is the negative energy cost, with a large penalty for any temperature limit violation.

The team uses a Deep Q-network with a target network and an experience replay buffer of about one million transitions, first trained in a simulator calibrated on two years of historical logs. Epsilon is annealed from 1.0 to 0.05 over the first 200,000 steps.

After deployment with a human override, the learned Q-function consistently ranks cautious setpoint changes higher during load spikes. Over a three-month trial, cooling energy falls by roughly 10 to 15 percent compared with the previous rule-based controller, without any temperature limit violations.